python /home/admin/mtr/script_for_cron.py -j default -m 20 -a 'python3 ~/workarea/git/Velours/python/prod/memo/memo.py --job=sla_per_day --raspi=20092 --dashboard_places=refus_amiens --last_days 7 -u_s -rs' -s sla_RUBBIA -M 0 -S 0 -U 90,80,95 import MySQLdb succeeded Import error (python version) Need env var : LOGRASPI Raspi : 20092, Date : _cr_cnf_ : /home/admin/mtr/.credentials/credentials.py _cr_cnf_ : /home/admin/mtr/.credentials/credentials.py warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 10383 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 10391 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10392 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 10387 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 10387 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 10390 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 10390 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11838 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11838 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 10386 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 10385 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 10385 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 10393 send_mail_cod have less outputs used (0) than in the step definition (1) : some outputs may be not used ! Step 10430 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 3 of step 10385 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 10383 doesn't seem to be define in the database( WARNING : type of input 2 of step 10387 doesn't seem to be define in the database( WARNING : output 1 of step 10383 have datatype=2 whereas input 1 of step 10390 have datatype=7 WARNING : type of output 2 of step 10390 doesn't seem to be define in the database( WARNING : type of input 1 of step 10384 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 10386 have datatype=10 whereas input 3 of step 10393 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10391 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of output 1 of step 10392 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of input 5 of step 10393 doesn't seem to be define in the database( WARNING : output 0 of step 11836 have datatype=11 whereas input 5 of step 10393 have datatype=None WARNING : output 0 of step 10386 have datatype=10 whereas input 0 of step 11836 have datatype=18 WARNING : type of output 3 of step 11838 doesn't seem to be define in the database( WARNING : type of input 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 2 of step 11838 doesn't seem to be define in the database( WARNING : output 1 of step 10384 have datatype=7 whereas input 2 of step 11838 have datatype=None WARNING : output 0 of step 10390 have datatype=1 whereas input 0 of step 10384 have datatype=2 DataTypes for each output/input checked ! Unexpected error treated as WARNING for now expected given param in final step {'name_pipeline': 'aggregate_carac_ratio', 'hashtag_type': 4213, 'hashtag_proportion': 'barquette_opaque,carton,ela,etiquette,film_plastique,kraft,metal,papier,pehd,pet_clair,pet_opaque,pet_fonce', 'hashtag_parmi': 'refus,textiles_sanitaires', 'hashtag_weights': {'barquette_opaque': 1, 'carton': 1, 'ela': 1, 'etiquette': 1, 'film_plastique': 1, 'kraft': 1, 'metal': 1, 'papier': 1, 'pehd': 1, 'pet_clair': 1, 'pet_opaque': 1, 'pet_fonce': 1, 'textiles_sanitaires': 1, 'refus': 3}} Found hashtag_parmi in final step, will be used and useful if different from matiere_majoritaire in classifier TODO : Insert select and so on # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 10383 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 10391 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10392 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 10387 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 10387 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 10390 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 10390 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11838 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11838 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 10386 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 10385 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 10385 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 10393 send_mail_cod have less outputs used (0) than in the step definition (1) : some outputs may be not used ! Step 10430 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 3 of step 10385 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 10383 doesn't seem to be define in the database( WARNING : type of input 2 of step 10387 doesn't seem to be define in the database( WARNING : output 1 of step 10383 have datatype=2 whereas input 1 of step 10390 have datatype=7 WARNING : type of output 2 of step 10390 doesn't seem to be define in the database( WARNING : type of input 1 of step 10384 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 10386 have datatype=10 whereas input 3 of step 10393 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10391 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of output 1 of step 10392 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of input 5 of step 10393 doesn't seem to be define in the database( WARNING : output 0 of step 11836 have datatype=11 whereas input 5 of step 10393 have datatype=None WARNING : output 0 of step 10386 have datatype=10 whereas input 0 of step 11836 have datatype=18 WARNING : type of output 3 of step 11838 doesn't seem to be define in the database( WARNING : type of input 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 2 of step 11838 doesn't seem to be define in the database( WARNING : output 1 of step 10384 have datatype=7 whereas input 2 of step 11838 have datatype=None WARNING : output 0 of step 10390 have datatype=1 whereas input 0 of step 10384 have datatype=2 DataTypes for each output/input checked ! Unexpected error treated as WARNING for now expected given param in final step {'name_pipeline': 'aggregate_carac_ratio', 'hashtag_type': 4213, 'hashtag_proportion': 'barquette_opaque,carton,ela,etiquette,film_plastique,kraft,metal,papier,pehd,pet_clair,pet_opaque,pet_fonce', 'hashtag_parmi': 'refus,textiles_sanitaires', 'hashtag_weights': {'barquette_opaque': 1, 'carton': 1, 'ela': 1, 'etiquette': 1, 'film_plastique': 1, 'kraft': 1, 'metal': 1, 'papier': 1, 'pehd': 1, 'pet_clair': 1, 'pet_opaque': 1, 'pet_fonce': 1, 'textiles_sanitaires': 1, 'refus': 3}} Found hashtag_parmi in final step, will be used and useful if different from matiere_majoritaire in classifier Fonction SLA_RUBBIA TODO : Insert select and so on ***** analysis of device with port 20092 for dashboard refus_amiens ***** **** analysis of day 2025/01/30/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! WARNING: No hour configured for port 20092, from 00:00 to 24:00 used 0:00:00 apple pause between two photos 15 Find filename in sqlite. error for extension of file /home/pi/Desktop/images/30012025/10/31/image_30012025_10_31_02_506380_0001.jpg~ Info from logs total number of images : 5636 coverage for 15 seconds : 23:29:00 (97.85%) coverage for 30 seconds : 23:30:00 (97.92%) max time between two photos : 0:29:00 results of pre-diagnostic : light ok 5025, not duplicated 5463, two criteria ok 4921, nb forced upload 0 end of day status of photos as found in sqllite no_longer_on_disk : 174 dark : 542 nothing_done : 1 ok_for_neural_classification : 4920 Unable to find info for dashboard number 15 for day 2025-01-30 06:31:57.918817 Info from dashboard total number of uploaded images : 0 number of useful images : 4921 number of finished photos : 0 (0.00% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 0 (0.00% of two criteria ok) **** analysis of day 2025/01/31/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! WARNING: No hour configured for port 20092, from 00:00 to 24:00 used 0:00:00 apple pause between two photos 15 Find filename in sqlite. Catched exception ! ERROR in database : database disk image is malformed sqllite_today : /data_2/data_log/log_raspi/2025/01/31/20092/sqlite.db ERROR in database : database disk image is malformed sqllite_today : /data_2/data_log/log_raspi/2025/01/31/20092/sqlite.db Info from logs total number of images : 0 coverage for 15 seconds : 0:00:00 (0.00%) coverage for 30 seconds : 0:00:00 (0.00%) max time between two photos : 0:00:00 results of pre-diagnostic : light ok 5458, not duplicated 5333, two criteria ok 5062, nb forced upload 0 end of day status of photos as found in sqllite Unable to find info for dashboard number 15 for day 2025-01-31 06:31:57.918817 Info from dashboard total number of uploaded images : 0 number of useful images : 5062 number of finished photos : 0 (0.00% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 0 (0.00% of two criteria ok) **** analysis of day 2025/02/01/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! WARNING: No hour configured for port 20092, from 00:00 to 24:00 used 0:00:00 apple pause between two photos 15 Find filename in sqlite. Info from logs total number of images : 5752 coverage for 15 seconds : 23:58:00 (99.86%) coverage for 30 seconds : 23:58:30 (99.90%) max time between two photos : 0:01:15 results of pre-diagnostic : light ok 5471, not duplicated 5502, two criteria ok 5229, nb forced upload 0 end of day status of photos as found in sqllite no_longer_on_disk : 258 dark : 273 nothing_done : 1 ok_for_neural_classification : 5220 Unable to find info for dashboard number 15 for day 2025-02-01 06:31:57.918817 Info from dashboard total number of uploaded images : 0 number of useful images : 5229 number of finished photos : 0 (0.00% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 0 (0.00% of two criteria ok) **** analysis of day 2025/02/02/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! WARNING: No hour configured for port 20092, from 00:00 to 24:00 used 0:00:00 apple pause between two photos 15 Find filename in sqlite. Info from logs total number of images : 5756 coverage for 15 seconds : 23:59:00 (99.93%) coverage for 30 seconds : 23:59:30 (99.97%) max time between two photos : 0:01:15 results of pre-diagnostic : light ok 5593, not duplicated 5584, two criteria ok 5431, nb forced upload 0 end of day status of photos as found in sqllite no_longer_on_disk : 172 dark : 153 nothing_done : 5 ok_for_neural_classification : 5426 Unable to find info for dashboard number 15 for day 2025-02-02 06:31:57.918817 Info from dashboard total number of uploaded images : 0 number of useful images : 5431 number of finished photos : 0 (0.00% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 0 (0.00% of two criteria ok) **** analysis of day 2025/02/03/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! file_take_pic not found at expected location (Did the log changed ? ) file_decision not found at expected location (Did the log changed ? ) VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! Error parsing crontab ! [Errno 2] No such file or directory: '' **** analysis of day 2025/02/04/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! file_take_pic not found at expected location (Did the log changed ? ) file_decision not found at expected location (Did the log changed ? ) VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! Error parsing crontab ! [Errno 2] No such file or directory: '' **** analysis of day 2025/02/05/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! file_take_pic not found at expected location (Did the log changed ? ) file_decision not found at expected location (Did the log changed ? ) VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! Error parsing crontab ! [Errno 2] No such file or directory: '' **** analysis for all days **** Info from logs total number of images : 17144 coverage for 0 seconds : 2 days, 23:26:00 (74.41%) coverage for 0 seconds : 2 days, 23:28:00 (74.45%) max time between two photos : 0:31:30 results of pre-diagnostic : light ok 21547, not duplicated 21882, two criteria ok 20643, nb forced upload 0 end of day status of photos as found in sqllite no_longer_on_disk : 604 dark : 968 nothing_done : 7 ok_for_neural_classification : 15566 Info from dashboard total number of uploaded images : 0 number of useful images : 20643 number of finished photos : 0 (0.00% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 0 (0.00% of two criteria ok) cvs resume : date,nb_photos,% time,nb ok,uploaded,to upload, % uploaded, nb useful, % completed photos, last_update, remark 2025/01/30,5636,97.85%,4921,0,4920,0.00%,0,0.00%,0000/00/00, 2025/01/31,0,0.00%,5062,0,0,0.00%,0,0.00%,0000/00/00, 2025/02/01,5752,99.86%,5229,0,5220,0.00%,0,0.00%,0000/00/00, 2025/02/02,5756,99.93%,5431,0,5426,0.00%,0,0.00%,0000/00/00, coverage for this period for every 0 second 2025/01/30/ 0.9784835472632785 2025/01/31/ 0.0 2025/02/01/ 0.998622669243857 2025/02/02/ 0.9993171217259459 2025/02/05/ 0.7441058345582704 mean value for this period : 0.7441058345582703 coverage for this period for every 0 second 2025/01/30/ 0.9791779997453675 2025/01/31/ 0.0 2025/02/01/ 0.9989698954849014 2025/02/02/ 0.9996643479669903 2025/02/05/ 0.7444530607993148 mean value for this period : 0.7444530607993148 2.50user 0.86system 0:11.03elapsed 30%CPU (0avgtext+0avgdata 194432maxresident)k 294984inputs+80outputs (647major+93729minor)pagefaults 0swaps