python /home/admin/mtr/script_for_cron.py -j default -m 20 -a 'python3 ~/workarea/git/Velours/python/prod/memo/memo.py --job=sla_per_day --raspi=30073 --dashboard_places=arc_en_ciel_jrm --last_days 7 -u_s' -s sla_RUBBIA -M 0 -S 0 -U 90,80,95 import MySQLdb succeeded Import error (python version) Need env var : LOGRASPI Raspi : 30073, Date : _cr_cnf_ : /home/admin/mtr/.credentials/credentials.py _cr_cnf_ : /home/admin/mtr/.credentials/credentials.py warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected error treated as WARNING for now expected given param in final step {'name_pipeline': 'aggregate_carac_ratio', 'hashtag_type': 3726, 'hashtag_proportion': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,kraft,metal,plastique', 'hashtag_parmi': 'papier,background', 'hashtag_weights': {'Carton_brun': 1.5, 'Carton_gris': 1.5, 'Teint_Dans_La_Masse': 1.0, 'autre_refus': 1.5, 'cartonnette': 1.0, 'kraft': 1.5, 'metal': 3, 'plastique': 1}} Found hashtag_parmi in final step, will be used and useful if different from matiere_majoritaire in classifier TODO : Insert select and so on # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected error treated as WARNING for now expected given param in final step {'name_pipeline': 'aggregate_carac_ratio', 'hashtag_type': 3726, 'hashtag_proportion': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,kraft,metal,plastique', 'hashtag_parmi': 'papier,background', 'hashtag_weights': {'Carton_brun': 1.5, 'Carton_gris': 1.5, 'Teint_Dans_La_Masse': 1.0, 'autre_refus': 1.5, 'cartonnette': 1.0, 'kraft': 1.5, 'metal': 3, 'plastique': 1}} Found hashtag_parmi in final step, will be used and useful if different from matiere_majoritaire in classifier Fonction SLA_RUBBIA TODO : Insert select and so on ***** analysis of device with port 30073 for dashboard arc_en_ciel_jrm ***** **** analysis of day 2025/08/21/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 8527 coverage for 10 seconds : 23:23:57 (97.50%) coverage for 20 seconds : 23:45:10 (98.97%) max time between two photos : 0:01:20 results of pre-diagnostic : light ok 8456, not duplicated 3089, two criteria ok 3089, nb forced upload 76 end of day status of photos as found in sqllite duplicated : 5366 nothing_done : 1 ok_for_neural_classification : 396 uploaded : 2764 Info from dashboard total number of uploaded images : 3427 number of useful images : 3360 number of finished photos : 2011 (59.85% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 1349 (40.15% of two criteria ok) number of finished batches : 39 (63.93% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 22 (36.07% of total) last update : 2025-08-22 17:27:30 **** analysis of day 2025/08/22/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 8539 coverage for 10 seconds : 23:23:22 (97.46%) coverage for 20 seconds : 23:47:06 (99.11%) max time between two photos : 0:01:21 results of pre-diagnostic : light ok 8447, not duplicated 2723, two criteria ok 2723, nb forced upload 97 end of day status of photos as found in sqllite duplicated : 5724 nothing_done : 1 ok_for_neural_classification : 4 uploaded : 2810 Info from dashboard total number of uploaded images : 2891 number of useful images : 2831 number of finished photos : 1313 (46.38% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 1518 (53.62% of two criteria ok) number of finished batches : 36 (54.55% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 30 (45.45% of total) last update : 2025-08-25 16:11:44 **** analysis of day 2025/08/23/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 2130 coverage for 10 seconds : 5:50:01 (100.00%) coverage for 20 seconds : 5:56:02 (100.00%) max time between two photos : 0:01:20 results of pre-diagnostic : light ok 2102, not duplicated 720, two criteria ok 720, nb forced upload 28 end of day status of photos as found in sqllite duplicated : 1382 nothing_done : 1 uploaded : 747 Info from dashboard total number of uploaded images : 751 number of useful images : 738 number of finished photos : 0 (0.00% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 738 (100.00% of two criteria ok) number of finished batches : 0 (0.00% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 15 (100.00% of total) last update : 2025-08-25 17:47:53 **** analysis of day 2025/08/24/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! No photos configured for this day pause between two photos 10 Unable to retrieve photo time from log, find in sqlite. Find filename in sqlite. Info from logs total number of images : 0 coverage for 10 seconds : 0:00:00 (-0.00%) coverage for 20 seconds : 0:00:00 (-0.00%) max time between two photos : 0:00:00 results of pre-diagnostic : light ok 0, not duplicated 0, two criteria ok 0, nb forced upload 0 end of day status of photos as found in sqllite Unable to find info for dashboard number 10 for day 2025-08-24 06:30:27.100791 Info from dashboard total number of uploaded images : 0 number of useful images : 0 **** analysis of day 2025/08/25/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 5:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 6755 coverage for 10 seconds : 18:32:31 (97.59%) coverage for 20 seconds : 18:49:01 (99.04%) max time between two photos : 0:01:21 results of pre-diagnostic : light ok 6694, not duplicated 2412, two criteria ok 2412, nb forced upload 61 end of day status of photos as found in sqllite duplicated : 4282 nothing_done : 1 uploaded : 2472 Info from dashboard total number of uploaded images : 2480 number of useful images : 2438 number of finished photos : 559 (22.93% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 1879 (77.07% of two criteria ok) number of finished batches : 16 (27.59% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 42 (72.41% of total) last update : 2025-08-26 18:52:23 **** analysis of day 2025/08/26/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 8539 coverage for 10 seconds : 23:26:52 (97.70%) coverage for 20 seconds : 23:47:10 (99.11%) max time between two photos : 0:01:21 results of pre-diagnostic : light ok 8458, not duplicated 2466, two criteria ok 2466, nb forced upload 86 end of day status of photos as found in sqllite duplicated : 5988 nothing_done : 1 uploaded : 2550 Info from dashboard total number of uploaded images : 2733 number of useful images : 2681 number of finished photos : 917 (34.20% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 1764 (65.80% of two criteria ok) number of finished batches : 30 (46.88% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 34 (53.12% of total) last update : 2025-08-27 17:37:33 **** analysis of day 2025/08/27/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 8561 coverage for 10 seconds : 23:31:10 (98.00%) coverage for 20 seconds : 23:51:01 (99.38%) max time between two photos : 0:01:21 results of pre-diagnostic : light ok 8470, not duplicated 3571, two criteria ok 3571, nb forced upload 91 end of day status of photos as found in sqllite duplicated : 4899 nothing_done : 1 ok_for_neural_classification : 5 uploaded : 3656 Info from dashboard total number of uploaded images : 3700 number of useful images : 3679 number of finished photos : 1289 (35.04% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 2390 (64.96% of two criteria ok) number of finished batches : 28 (36.36% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 49 (63.64% of total) last update : 2025-08-28 06:27:49 **** analysis for all days **** Info from logs total number of images : 43051 coverage for 10 seconds : 4 days, 22:07:53 (98.46%) coverage for 20 seconds : 4 days, 23:55:30 (99.95%) max time between two photos : 0:08:04 results of pre-diagnostic : light ok 42627, not duplicated 14981, two criteria ok 14981, nb forced upload 439 end of day status of photos as found in sqllite duplicated : 27641 nothing_done : 6 ok_for_neural_classification : 405 uploaded : 14999 Info from dashboard total number of uploaded images : 15982 number of useful images : 15727 number of finished photos : 6089 (38.72% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 9638 (61.28% of two criteria ok) number of finished batches : 149 (43.70% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 192 (56.30% of total) last update : 2025-08-28 06:27:49 You can launch datou with following url to update dashboard https://www.fotonower.com/api/v1/datou/current/set?mtr_portfolio_id=26103981,26125978,26164221,26190656,26218397,26247250&mtr_datou_id=4878 cvs resume : date,nb_photos,% time,nb ok,uploaded,to upload, % uploaded, nb useful, % completed photos, last_update, remark 2025/08/21,8527,97.50%,3089,2764,396,87.47%,3360,59.85%,2025/08/22, 2025/08/22,8539,97.46%,2723,2810,4,99.86%,2831,46.38%,2025/08/25, 2025/08/23,2130,100.00%,720,747,0,100.00%,738,0.00%,2025/08/25, 2025/08/24,0,-0.00%,0,0,0,0.00%,0,0.00%,0000/00/00, 2025/08/25,6755,97.59%,2412,2472,0,100.00%,2438,22.93%,2025/08/26, 2025/08/26,8539,97.70%,2466,2550,0,100.00%,2681,34.20%,2025/08/27, 2025/08/27,8561,98.00%,3571,3656,5,99.86%,3679,35.04%,2025/08/28, coverage for this period for every 10 second 2025/08/21/ 0.9749765622287295 2025/08/22/ 0.974571464947511 2025/08/23/ 1.0 2025/08/24/ -0.0 2025/08/25/ 0.9759060804982529 2025/08/26/ 0.9770020486348222 2025/08/27/ 0.984578662480842 mean value for this period : 0.8410049741128797 coverage for this period for every 20 second 2025/08/21/ 0.9897105290570493 2025/08/22/ 0.9910531371890878 2025/08/23/ 1.0 2025/08/24/ -0.0 2025/08/25/ 0.9903799763154432 2025/08/26/ 0.9910994340212271 2025/08/27/ 0.9995277056216922 mean value for this period : 0.8516815403149284 3.06user 0.88system 0:18.27elapsed 21%CPU (0avgtext+0avgdata 178768maxresident)k 229768inputs+80outputs (16major+87369minor)pagefaults 0swaps