python /home/admin/mtr/script_for_cron.py -j default -m 20 -a 'python3 ~/workarea/git/Velours/python/prod/memo/memo.py --job=sla_per_day --raspi=30073 --dashboard_places=arc_en_ciel_jrm --last_days 7 -u_s' -s sla_RUBBIA -M 0 -S 0 -U 90,80,95 import MySQLdb succeeded Import error (python version) Need env var : LOGRASPI Raspi : 30073, Date : _cr_cnf_ : /home/admin/mtr/.credentials/credentials.py _cr_cnf_ : /home/admin/mtr/.credentials/credentials.py warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected error treated as WARNING for now expected given param in final step {'name_pipeline': 'aggregate_carac_ratio', 'hashtag_type': 3726, 'hashtag_proportion': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,kraft,metal,plastique', 'hashtag_parmi': 'papier,background', 'hashtag_weights': {'Carton_brun': 1.5, 'Carton_gris': 1.5, 'Teint_Dans_La_Masse': 1.0, 'autre_refus': 1.5, 'cartonnette': 1.0, 'kraft': 1.5, 'metal': 3, 'plastique': 1}} Found hashtag_parmi in final step, will be used and useful if different from matiere_majoritaire in classifier TODO : Insert select and so on # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 13596 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11881 final have less inputs used (1) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11881 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13625 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 11884 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 13596 have datatype=10 whereas input 3 of step 11884 have datatype=6 WARNING : type of output 1 of step 13596 doesn't seem to be define in the database( WARNING : type of input 3 of step 11881 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 13596 have datatype=10 whereas input 0 of step 13625 have datatype=18 WARNING : type of output 1 of step 13625 doesn't seem to be define in the database( WARNING : type of input 5 of step 11884 doesn't seem to be define in the database( DataTypes for each output/input checked ! Unexpected error treated as WARNING for now expected given param in final step {'name_pipeline': 'aggregate_carac_ratio', 'hashtag_type': 3726, 'hashtag_proportion': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,kraft,metal,plastique', 'hashtag_parmi': 'papier', 'hashtag_weights': {'Carton_brun': 2, 'Carton_gris': 1.5, 'Teint_Dans_La_Masse': 1.0, 'autre_refus': 1.5, 'cartonnette': 1.0, 'kraft': 1.5, 'metal': 3, 'plastique': 1}} Found hashtag_parmi in final step, will be used and useful if different from matiere_majoritaire in classifier Fonction SLA_RUBBIA TODO : Insert select and so on ***** analysis of device with port 30073 for dashboard arc_en_ciel_jrm ***** **** analysis of day 2025/09/08/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 5:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 6779 coverage for 10 seconds : 18:38:15 (98.09%) coverage for 20 seconds : 18:52:59 (99.39%) max time between two photos : 0:01:21 results of pre-diagnostic : light ok 6710, not duplicated 3113, two criteria ok 3113, nb forced upload 69 end of day status of photos as found in sqllite duplicated : 3597 nothing_done : 1 ok_for_neural_classification : 4 uploaded : 3177 Info from dashboard total number of uploaded images : 3547 number of useful images : 3523 number of finished photos : 3523 (100.00% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 0 (0.00% of two criteria ok) number of finished batches : 54 (100.00% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 0 (0.00% of total) last update : 2025-09-09 07:10:38 **** analysis of day 2025/09/09/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 8568 coverage for 10 seconds : 23:32:48 (98.11%) coverage for 20 seconds : 23:51:59 (99.44%) max time between two photos : 0:01:21 results of pre-diagnostic : light ok 8496, not duplicated 4082, two criteria ok 4082, nb forced upload 72 end of day status of photos as found in sqllite duplicated : 4414 nothing_done : 1 ok_for_neural_classification : 10 uploaded : 4143 Info from dashboard total number of uploaded images : 5027 number of useful images : 4990 number of finished photos : 4990 (100.00% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 0 (0.00% of two criteria ok) number of finished batches : 67 (100.00% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 0 (0.00% of total) last update : 2025-09-10 07:11:27 **** analysis of day 2025/09/10/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 8575 coverage for 10 seconds : 23:31:20 (98.01%) coverage for 20 seconds : 23:53:03 (99.52%) max time between two photos : 0:01:21 results of pre-diagnostic : light ok 8480, not duplicated 1245, two criteria ok 1245, nb forced upload 100 end of day status of photos as found in sqllite duplicated : 7232 nothing_done : 1 ok_for_neural_classification : 3 uploaded : 1339 Info from dashboard total number of uploaded images : 1566 number of useful images : 1421 number of finished photos : 1421 (100.00% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 0 (0.00% of two criteria ok) number of finished batches : 27 (100.00% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 0 (0.00% of total) last update : 2025-09-11 11:10:36 **** analysis of day 2025/09/11/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 8574 coverage for 10 seconds : 23:33:20 (98.15%) coverage for 20 seconds : 23:52:56 (99.51%) max time between two photos : 0:01:21 results of pre-diagnostic : light ok 8483, not duplicated 3502, two criteria ok 3502, nb forced upload 91 end of day status of photos as found in sqllite duplicated : 4981 nothing_done : 1 ok_for_neural_classification : 3 uploaded : 3589 Info from dashboard total number of uploaded images : 3985 number of useful images : 3941 number of finished photos : 2711 (68.79% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 1230 (31.21% of two criteria ok) number of finished batches : 41 (67.21% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 20 (32.79% of total) last update : 2025-09-12 13:55:39 **** analysis of day 2025/09/12/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 8550 coverage for 10 seconds : 23:27:42 (97.76%) coverage for 20 seconds : 23:49:01 (99.24%) max time between two photos : 0:01:21 results of pre-diagnostic : light ok 8465, not duplicated 4941, two criteria ok 4941, nb forced upload 85 end of day status of photos as found in sqllite duplicated : 3524 nothing_done : 1 ok_for_neural_classification : 5 uploaded : 5020 Info from dashboard total number of uploaded images : 5506 number of useful images : 5488 number of finished photos : 3427 (62.45% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 2061 (37.55% of two criteria ok) number of finished batches : 48 (60.00% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 32 (40.00% of total) last update : 2025-09-13 07:35:27 **** analysis of day 2025/09/13/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 1784 coverage for 10 seconds : 4:53:13 (97.74%) coverage for 20 seconds : 4:58:10 (99.39%) max time between two photos : 0:01:20 results of pre-diagnostic : light ok 1774, not duplicated 784, two criteria ok 784, nb forced upload 15 end of day status of photos as found in sqllite duplicated : 989 nothing_done : 1 uploaded : 794 Info from dashboard total number of uploaded images : 926 number of useful images : 921 number of finished photos : 921 (100.00% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 0 (0.00% of two criteria ok) number of finished batches : 15 (100.00% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 0 (0.00% of total) last update : 2025-09-14 07:00:27 **** analysis of day 2025/09/14/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! file_take_pic not found at expected location (Did the log changed ? ) file_decision not found at expected location (Did the log changed ? ) VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! Error parsing crontab ! [Errno 2] No such file or directory: '' **** analysis for all days **** Info from logs total number of images : 42830 coverage for 0 seconds : 4 days, 21:36:38 (98.01%) coverage for 0 seconds : 4 days, 23:18:08 (99.42%) max time between two photos : 0:08:05 results of pre-diagnostic : light ok 42408, not duplicated 17667, two criteria ok 17667, nb forced upload 432 end of day status of photos as found in sqllite duplicated : 24737 nothing_done : 6 ok_for_neural_classification : 25 uploaded : 18062 Info from dashboard total number of uploaded images : 20557 number of useful images : 20284 number of finished photos : 16993 (83.78% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 3291 (16.22% of two criteria ok) number of finished batches : 252 (82.89% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 52 (17.11% of total) last update : 2025-09-14 07:00:27 You can launch datou with following url to update dashboard https://www.fotonower.com/api/v1/datou/current/set?mtr_portfolio_id=26734743,26769748&mtr_datou_id=4878 cvs resume : date,nb_photos,% time,nb ok,uploaded,to upload, % uploaded, nb useful, % completed photos, last_update, remark 2025/09/08,6779,98.09%,3113,3177,4,99.87%,3523,100.00%,2025/09/09, 2025/09/09,8568,98.11%,4082,4143,10,99.76%,4990,100.00%,2025/09/10, 2025/09/10,8575,98.01%,1245,1339,3,99.78%,1421,100.00%,2025/09/11, 2025/09/11,8574,98.15%,3502,3589,3,99.92%,3941,68.79%,2025/09/12, 2025/09/12,8550,97.76%,4941,5020,5,99.90%,5488,62.45%,2025/09/13, 2025/09/13,1784,97.74%,784,794,0,100.00%,921,100.00%,2025/09/14, coverage for this period for every 0 second 2025/09/08/ 0.9809353937923069 2025/09/09/ 0.9811224666952164 2025/09/10/ 0.9801039363881526 2025/09/11/ 0.9814928413523305 2025/09/12/ 0.9775807590365629 2025/09/13/ 0.977443191288405 2025/09/14/ 0.9801015754848447 mean value for this period : 0.9798257377196885 coverage for this period for every 0 second 2025/09/08/ 0.9938595593502829 2025/09/09/ 0.9944443801432887 2025/09/10/ 0.9951851294575169 2025/09/11/ 0.9951041100012732 2025/09/12/ 0.9923841711130916 2025/09/13/ 0.9939441080060003 2025/09/14/ 0.9941989935045394 mean value for this period : 0.9941600645108561 2.77user 0.86system 0:10.82elapsed 33%CPU (0avgtext+0avgdata 178620maxresident)k 271928inputs+72outputs (4major+84849minor)pagefaults 0swaps