python /home/admin/mtr/script_for_cron.py -j default -m 20 -a 'python3 ~/workarea/git/Velours/python/prod/memo/memo.py --job=sla_per_day --raspi=20084 --dashboard_places=dechet_Valcor_v2 --last_days 7 -u_s' -s sla_RUBBIA -M 0 -S 0 -U 90,80,95 import MySQLdb succeeded Import error (python version) Need env var : LOGRASPI Raspi : 20084, Date : _cr_cnf_ : /home/admin/mtr/.credentials/credentials.py _cr_cnf_ : /home/admin/mtr/.credentials/credentials.py warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11939 matching_dashboard is not consistent : 1 used against 0 in the step definition ! Step 11940 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 0 of step 11939 doesn't seem to be define in the database( WARNING : type of input 2 of step 11940 doesn't seem to be define in the database( DataTypes for each output/input checked ! We expected an STS datou with a camera_place_name, so there is a PB, datou_id: 4311 ! None was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? {'id': '4311', 'name': 'sts_dechet_valcor_matching_v2_2022Q3', 'limit_max': 100, 'nb_steps': 2, 'estimated_time': 8640, 'list_steps': [{'id': 11939, 'type': 'matching_dashboard', 'param': '0', 'json_param_as_string': "{'camera_matching': 'plaque_Valcor', 'max_time_diff': 120, 'matching_method': '1_vs_n', 'input_default_values': []}", 'json_param': {'camera_matching': 'plaque_Valcor', 'max_time_diff': 120, 'matching_method': '1_vs_n', 'input_default_values': []}, 'nb_input': 1, 'nb_output': 0, 'prod': False, 'is_local': False, 'is_datou_depend': False, 'is_photo_id_local': False, 'list_data': [{'step_id': 11939, 'datou_id': 4311, 'type_id': 18, 'name': 'portfolio_id', 'output_or_input': False, 'data_order_id': 0, 'type_python': 'int'}]}, {'id': 11940, 'type': 'split_time_score', 'param': '0', 'json_param_as_string': "{'camera_place_name': 'dechet_Valcor_v2', 'ETA': 8640, 'param_moyenne_mobile': 5, 'thcl': 3442, 'list_task_complete': [''], 'list_tasks': 'LOAD,ORDER,CONVENTION,SPLIT_TRUCK,merge_port_according_to_plate_info,remove_no_plate_batch,CREATE_PORT_LAUNCH_CARAC_FILL_DASHBOARD', 'forced_hashtag': 'truck', 'list_hashtag_class_create': 'truck', 'ponderation': '0.001,0.001,0.001,0.001,0.001', 'host': 'www.fotonower.com', 'datou_proportion_papier': {'test': 370, 'truck': 4234}, 'datou_matching_id': 4311, 'cod': 1, 'min_number_photo_per_batch': 2, 'datou_truck_for_info': 4234}", 'json_param': {'camera_place_name': 'dechet_Valcor_v2', 'ETA': 8640, 'param_moyenne_mobile': 5, 'thcl': 3442, 'list_task_complete': [''], 'list_tasks': 'LOAD,ORDER,CONVENTION,SPLIT_TRUCK,merge_port_according number of started photos : 0 (0.00% of two criteria ok) number of finished batches : 74 (100.00% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 0 (0.00% of total) last update : 2025-05-23 11:17:15 **** analysis of day 2025/05/22/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! **** analysis of day 2025/05/23/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 8526 coverage for 10 seconds : 23:07:33 (96.36%) coverage for 20 seconds : 23:45:20 (98.98%) max time between two photos : 0:01:20 results of pre-diagnostic : light ok 8532, not duplicated 7259, two criteria ok 7259, nb forced upload 0 end of day status of photos as found in sqllite duplicated : 1270 nothing_done : 13 ok_for_neural_classification : 1963 uploaded : 5280 Info from dashboard total number of uploaded images : 7380 number of useful images : 7259 number of finished photos : 277 (3.82% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 0 (0.00% of two criteria ok) number of finished batches : 14 (100.00% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 0 (0.00% of total) last update : 2025-05-24 06:01:27 **** analysis of day 2025/05/24/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 8517 coverage for 10 seconds : 23:04:04 (96.12%) coverage for 20 seconds : 23:43:35 (98.86%) max time between two photos : 0:01:20 results of pre-diagnostic : light ok 8532, not duplicated 7866, two criteria ok 7866, nb forced upload 0 end of day status of photos as found in sqllite duplicated : 664 nothing_done : 4 ok_for_neural_classification : 1357 uploaded : 6492 Unable to find info for batches (last_run_id 1841754) in MTRPhoto.dashboard_results Info from dashboard total number of uploaded images : 7982 number of useful images : 7866 number of finished photos : 0 (0.00% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 0 (0.00% of two criteria ok) **** analysis of day 2025/05/25/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! **** analysis of day 2025/05/26/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Catched exception ! ERROR in database : database disk image is malformed sqllite_today : /data_2/data_log/log_raspi/2025/05/26/20084/sqlite.db ERROR in database : database disk image is malformed sqllite_today : /data_2/data_log/log_raspi/2025/05/26/20084/sqlite.db Info from logs total number of images : 0 coverage for 10 seconds : 0:00:00 (0.00%) coverage for 20 seconds : 0:00:00 (0.00%) max time between two photos : 0:00:00 results of pre-diagnostic : light ok 8532, not duplicated 7765, two criteria ok 7765, nb forced upload 0 end of day status of photos as found in sqllite Info from dashboard total number of uploaded images : 7874 number of useful images : 7765 number of finished photos : 711 (9.16% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 394 (5.07% of two criteria ok) number of finished batches : 133 (76.44% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 41 (23.56% of total) last update : 2025-05-27 06:18:48 **** analysis for all days **** Info from logs total number of images : 34061 coverage for 10 seconds : 3 days, 20:16:56 (76.90%) coverage for 20 seconds : 3 days, 22:53:34 (79.08%) max time between two photos : 0:06:39 results of pre-diagnostic : light ok 42631, not duplicated 37295, two criteria ok 37295, nb forced upload 0 end of day status of photos as found in sqllite duplicated : 4560 nothing_done : 31 ok_for_neural_classification : 3837 uploaded : 25632 no_longer_on_disk : 1 Info from dashboard total number of uploaded images : 37879 number of useful images : 37295 number of finished photos : 1826 (4.90% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 1196 (3.21% of two criteria ok) number of finished batches : 281 (64.90% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 152 (35.10% of total) last update : 2025-05-27 06:18:48 You can launch datou with following url to update dashboard https://www.fotonower.com/api/v1/datou/current/set?mtr_portfolio_id=23149689,23319659&mtr_datou_id=4311 cvs resume : date,nb_photos,% time,nb ok,uploaded,to upload, % uploaded, nb useful, % completed photos, last_update, remark 2025/05/20,8493,95.70%,7107,6792,300,95.77%,1114,28.01%,2025/05/22, 2025/05/21,8525,96.34%,7298,7068,217,97.02%,526,100.00%,2025/05/23, 2025/05/23,8526,96.36%,7259,5280,1963,72.90%,277,100.00%,2025/05/24, 2025/05/24,8517,96.12%,7866,6492,1357,82.71%,0,0.00%,2025/05/25, 2025/05/26,0,0.00%,7765,0,0,0.00%,1105,64.34%,2025/05/27, coverage for this period for every 10 second 2025/05/20/ 0.9570249655667311 2025/05/21/ 0.963356057361775 2025/05/23/ 0.9635875415224713 2025/05/24/ 0.961168532043195 2025/05/26/ 0.7690274192988344 mean value for this period : 0.9228329031586014 coverage for this period for every 20 second 2025/05/20/ 0.9859952082778736 2025/05/21/ 0.989479044896353 2025/05/23/ 0.9898262711373974 2025/05/24/ 0.9886109792937419 2025/05/26/ 0.7907823007210731 mean value for this period : 0.9489387608652878 2.61user 0.93system 0:13.52elapsed 26%CPU (0avgtext+0avgdata 194556maxresident)k 229776inputs+104outputs (82major+93796minor)pagefaults 0swaps