python /home/admin/mtr/script_for_cron.py -j default -m 20 -a 'python3 ~/workarea/git/Velours/python/prod/memo/memo.py --job=sla_per_day --raspi=30073 --dashboard_places=arc_en_ciel_jrm --last_days 7 -u_s' -s sla_RUBBIA -M 0 -S 0 -U 90,80,95 import MySQLdb succeeded Import error (python version) Need env var : LOGRASPI Raspi : 30073, Date : _cr_cnf_ : /home/admin/mtr/.credentials/credentials.py _cr_cnf_ : /home/admin/mtr/.credentials/credentials.py warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected error treated as WARNING for now expected given param in final step {'name_pipeline': 'aggregate_carac_ratio', 'hashtag_type': 3726, 'hashtag_proportion': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,kraft,metal,plastique', 'hashtag_parmi': 'papier', 'hashtag_weights': {'Carton_brun': 1.5, 'Carton_gris': 1.5, 'Teint_Dans_La_Masse': 1.0, 'autre_refus': 1.5, 'cartonnette': 1.0, 'kraft': 1.5, 'metal': 3, 'plastique': 1}} Found hashtag_parmi in final step, will be used and useful if different from matiere_majoritaire in classifier TODO : Insert select and so on # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 13596 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11881 final have less inputs used (1) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11881 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13625 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 11884 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 13596 have datatype=10 whereas input 3 of step 11884 have datatype=6 WARNING : type of output 1 of step 13596 doesn't seem to be define in the database( WARNING : type of input 3 of step 11881 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 13596 have datatype=10 whereas input 0 of step 13625 have datatype=18 WARNING : type of output 1 of step 13625 doesn't seem to be define in the database( WARNING : type of input 5 of step 11884 doesn't seem to be define in the database( DataTypes for each output/input checked ! Unexpected error treated as WARNING for now expected given param in final step {'name_pipeline': 'aggregate_carac_ratio', 'hashtag_type': 3726, 'hashtag_proportion': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,kraft,metal,plastique', 'hashtag_parmi': 'papier', 'hashtag_weights': {'Carton_brun': 2, 'Carton_gris': 1.5, 'Teint_Dans_La_Masse': 1.0, 'autre_refus': 1.5, 'cartonnette': 1.0, 'kraft': 1.5, 'metal': 3, 'plastique': 1}} Found hashtag_parmi in final step, will be used and useful if different from matiere_majoritaire in classifier Fonction SLA_RUBBIA TODO : Insert select and so on ***** analysis of device with port 30073 for dashboard arc_en_ciel_jrm ***** **** analysis of day 2025/01/27/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 5:00:00 apple pause between two photos 15 Find filename in sqlite. Info from logs total number of images : 4509 coverage for 15 seconds : 18:35:05 (97.82%) coverage for 30 seconds : 18:52:03 (99.30%) max time between two photos : 0:01:30 results of pre-diagnostic : light ok 4227, not duplicated 2524, two criteria ok 2524, nb forced upload 43 end of day status of photos as found in sqllite duplicated : 1703 nothing_done : 240 uploaded : 2566 Info from dashboard total number of uploaded images : 4066 number of useful images : 4052 number of finished photos : 3769 (93.02% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 283 (6.98% of two criteria ok) number of finished batches : 60 (93.75% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 4 (6.25% of total) last update : 2025-01-29 06:03:36 **** analysis of day 2025/01/28/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 1 Find filename in sqlite. Info from logs total number of images : 28304 coverage for 1 seconds : 7:51:44 (32.76%) coverage for 2 seconds : 9:26:53 (39.37%) max time between two photos : 0:01:30 results of pre-diagnostic : light ok 18137, not duplicated 2997, two criteria ok 2985, nb forced upload 251 end of day status of photos as found in sqllite no_longer_on_disk : 1 dark : 12 duplicated : 15661 nothing_done : 9524 uploaded : 3106 Info from dashboard total number of uploaded images : 8023 number of useful images : 8006 number of finished photos : 3264 (40.77% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 4742 (59.23% of two criteria ok) number of finished batches : 61 (70.93% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 25 (29.07% of total) last update : 2025-02-01 18:26:13 **** analysis of day 2025/01/29/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 45272 coverage for 10 seconds : 13:19:23 (55.51%) coverage for 20 seconds : 13:20:38 (55.60%) max time between two photos : 0:01:06 results of pre-diagnostic : light ok 47183, not duplicated 15402, two criteria ok 15359, nb forced upload 488 end of day status of photos as found in sqllite dark : 38 duplicated : 28153 nothing_done : 5207 ok_for_neural_classification : 1156 uploaded : 10718 Info from dashboard total number of uploaded images : 12969 number of useful images : 15359 number of finished photos : 6234 (40.59% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 6730 (43.82% of two criteria ok) number of finished batches : 28 (52.83% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 25 (47.17% of total) last update : 2025-02-02 06:12:15 **** analysis of day 2025/01/30/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 7826 coverage for 10 seconds : 21:29:09 (89.53%) coverage for 20 seconds : 21:47:55 (90.83%) max time between two photos : 0:01:20 results of pre-diagnostic : light ok 11184, not duplicated 2837, two criteria ok 2831, nb forced upload 80 end of day status of photos as found in sqllite duplicated : 4831 nothing_done : 360 uploaded : 2635 Info from dashboard total number of uploaded images : 3057 number of useful images : 3020 number of finished photos : 2411 (79.83% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 609 (20.17% of two criteria ok) number of finished batches : 54 (84.38% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 10 (15.62% of total) last update : 2025-02-01 05:59:06 **** analysis of day 2025/01/31/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 8562 coverage for 10 seconds : 23:29:40 (97.89%) coverage for 20 seconds : 23:51:03 (99.38%) max time between two photos : 0:01:21 results of pre-diagnostic : light ok 8468, not duplicated 3258, two criteria ok 3258, nb forced upload 100 end of day status of photos as found in sqllite duplicated : 5136 nothing_done : 359 uploaded : 3067 Info from dashboard total number of uploaded images : 3352 number of useful images : 3258 number of finished photos : 2780 (85.33% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 466 (14.30% of two criteria ok) number of finished batches : 47 (87.04% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 7 (12.96% of total) last update : 2025-02-01 08:09:15 **** analysis of day 2025/02/01/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 1778 coverage for 10 seconds : 4:52:50 (97.62%) coverage for 20 seconds : 4:57:11 (99.07%) max time between two photos : 0:01:20 results of pre-diagnostic : light ok 2121, not duplicated 1094, two criteria ok 1094, nb forced upload 15 end of day status of photos as found in sqllite duplicated : 868 nothing_done : 1 uploaded : 909 Info from dashboard total number of uploaded images : 940 number of useful images : 1094 number of finished photos : 790 (72.21% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 146 (13.35% of two criteria ok) number of finished batches : 13 (76.47% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 4 (23.53% of total) last update : 2025-02-02 07:17:03 **** analysis of day 2025/02/02/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! file_take_pic not found at expected location (Did the log changed ? ) file_decision not found at expected location (Did the log changed ? ) VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! Error parsing crontab ! [Errno 2] No such file or directory: '' **** analysis for all days **** Info from logs total number of images : 96251 coverage for 0 seconds : 3 days, 17:37:51 (74.69%) coverage for 0 seconds : 3 days, 20:15:43 (76.89%) max time between two photos : 0:08:07 results of pre-diagnostic : light ok 91320, not duplicated 28112, two criteria ok 28051, nb forced upload 977 end of day status of photos as found in sqllite duplicated : 56352 nothing_done : 15691 uploaded : 23001 no_longer_on_disk : 1 dark : 50 ok_for_neural_classification : 1156 Info from dashboard total number of uploaded images : 32407 number of useful images : 32224 number of finished photos : 19248 (59.73% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 12976 (40.27% of two criteria ok) number of finished batches : 263 (77.81% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 75 (22.19% of total) last update : 2025-02-02 07:17:03 You can launch datou with following url to update dashboard https://www.fotonower.com/api/v1/datou/current/set?mtr_portfolio_id=19946267,19990455,20032448,20054117,20090623,20128731&mtr_datou_id=4878 cvs resume : date,nb_photos,% time,nb ok,uploaded,to upload, % uploaded, nb useful, % completed photos, last_update, remark 2025/01/27,4509,97.82%,2524,2566,0,100.00%,4052,93.02%,2025/01/29, 2025/01/28,28304,32.76%,2985,3106,0,100.00%,8006,40.77%,2025/02/01, 2025/01/29,45272,55.51%,15359,10718,1156,90.26%,12964,48.09%,2025/02/02, 2025/01/30,7826,89.53%,2831,2635,0,100.00%,3020,79.83%,2025/02/01, 2025/01/31,8562,97.89%,3258,3067,0,100.00%,3246,85.64%,2025/02/01, 2025/02/01,1778,97.62%,1094,909,0,100.00%,936,84.40%,2025/02/02, coverage for this period for every 0 second 2025/01/27/ 0.9781575754031492 2025/01/28/ 0.32759638421740994 2025/01/29/ 0.5551337399738423 2025/01/30/ 0.8952534172849222 2025/01/31/ 0.9789465155846712 2025/02/01/ 0.9761653425190289 2025/02/02/ 0.7469339851942388 mean value for this period : 0.7797409943110374 coverage for this period for every 0 second 2025/01/27/ 0.9930408339303206 2025/01/28/ 0.39367353788817 2025/01/29/ 0.5560018055764534 2025/01/30/ 0.9082859755321242 2025/01/31/ 0.993796224493339 2025/02/01/ 0.99066614811934 2025/02/02/ 0.7688602156511433 mean value for this period : 0.8006178201701272 3.47user 1.73system 0:26.35elapsed 19%CPU (0avgtext+0avgdata 257104maxresident)k 224616inputs+80outputs (432major+158514minor)pagefaults 0swaps