python /home/admin/mtr/script_for_cron.py -j default -m 20 -a 'python3 ~/workarea/git/Velours/python/prod/memo/memo.py --job=sla_per_day --raspi=30073 --dashboard_places=arc_en_ciel_jrm --last_days 7 -u_s' -s sla_RUBBIA -M 0 -S 0 -U 90,80,95 import MySQLdb succeeded Import error (python version) Need env var : LOGRASPI Raspi : 30073, Date : _cr_cnf_ : /home/admin/mtr/.credentials/credentials.py _cr_cnf_ : /home/admin/mtr/.credentials/credentials.py warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected error treated as WARNING for now expected given param in final step {'name_pipeline': 'aggregate_carac_ratio', 'hashtag_type': 3726, 'hashtag_proportion': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,kraft,metal,plastique', 'hashtag_parmi': 'papier,background', 'hashtag_weights': {'Carton_brun': 1.5, 'Carton_gris': 1.5, 'Teint_Dans_La_Masse': 1.0, 'autre_refus': 1.5, 'cartonnette': 1.0, 'kraft': 1.5, 'metal': 3, 'plastique': 1}} Found hashtag_parmi in final step, will be used and useful if different from matiere_majoritaire in classifier TODO : Insert select and so on # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 13596 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11881 final have less inputs used (1) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11881 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13625 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 11884 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 13596 have datatype=10 whereas input 3 of step 11884 have datatype=6 WARNING : type of output 1 of step 13596 doesn't seem to be define in the database( WARNING : type of input 3 of step 11881 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 13596 have datatype=10 whereas input 0 of step 13625 have datatype=18 WARNING : type of output 1 of step 13625 doesn't seem to be define in the database( WARNING : type of input 5 of step 11884 doesn't seem to be define in the database( DataTypes for each output/input checked ! Unexpected error treated as WARNING for now expected given param in final step {'name_pipeline': 'aggregate_carac_ratio', 'hashtag_type': 3726, 'hashtag_proportion': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,kraft,metal,plastique', 'hashtag_parmi': 'papier', 'hashtag_weights': {'Carton_brun': 2, 'Carton_gris': 1.5, 'Teint_Dans_La_Masse': 1.0, 'autre_refus': 1.5, 'cartonnette': 1.0, 'kraft': 1.5, 'metal': 3, 'plastique': 1}} Found hashtag_parmi in final step, will be used and useful if different from matiere_majoritaire in classifier Fonction SLA_RUBBIA TODO : Insert select and so on ***** analysis of device with port 30073 for dashboard arc_en_ciel_jrm ***** **** analysis of day 2025/03/21/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 8527 coverage for 10 seconds : 23:22:04 (97.37%) coverage for 20 seconds : 23:45:13 (98.97%) max time between two photos : 0:01:21 results of pre-diagnostic : light ok 8449, not duplicated 4148, two criteria ok 4148, nb forced upload 83 end of day status of photos as found in sqllite duplicated : 4300 nothing_done : 1 uploaded : 4226 Info from dashboard total number of uploaded images : 4506 number of useful images : 4440 number of finished photos : 1801 (40.56% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 2639 (59.44% of two criteria ok) number of finished batches : 32 (45.71% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 38 (54.29% of total) last update : 2025-03-25 17:19:28 **** analysis of day 2025/03/22/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not don results of pre-diagnostic : light ok 8451, not duplicated 2602, two criteria ok 2602, nb forced upload 99 end of day status of photos as found in sqllite duplicated : 5849 nothing_done : 1 uploaded : 2700 Info from dashboard total number of uploaded images : 2736 number of useful images : 2632 number of finished photos : 2632 (100.00% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 0 (0.00% of two criteria ok) number of finished batches : 44 (100.00% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 0 (0.00% of total) last update : 2025-03-26 07:55:46 **** analysis of day 2025/03/26/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 8545 coverage for 10 seconds : 23:28:06 (97.79%) coverage for 20 seconds : 23:48:10 (99.18%) max time between two photos : 0:01:21 results of pre-diagnostic : light ok 8450, not duplicated 3905, two criteria ok 3905, nb forced upload 100 end of day status of photos as found in sqllite duplicated : 4540 nothing_done : 1 ok_for_neural_classification : 6 uploaded : 3998 Info from dashboard total number of uploaded images : 4026 number of useful images : 3967 number of finished photos : 3261 (82.20% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 706 (17.80% of two criteria ok) number of finished batches : 53 (81.54% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 12 (18.46% of total) last update : 2025-03-27 07:01:28 **** analysis of day 2025/03/27/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 8563 coverage for 10 seconds : 23:31:11 (98.00%) coverage for 20 seconds : 23:51:06 (99.38%) max time between two photos : 0:01:21 results of pre-diagnostic : light ok 8491, not duplicated 4285, two criteria ok 4285, nb forced upload 77 end of day status of photos as found in sqllite duplicated : 4204 nothing_done : 1 uploaded : 4358 Info from dashboard total number of uploaded images : 4388 number of useful images : 4336 number of finished photos : 4265 (98.36% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 71 (1.64% of two criteria ok) number of finished batches : 70 (98.59% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 1 (1.41% of total) last update : 2025-03-28 06:19:03 **** analysis for all days **** Info from logs total number of images : 42729 coverage for 10 seconds : 4 days, 21:16:25 (97.73%) coverage for 20 seconds : 4 days, 23:01:22 (99.19%) max time between two photos : 0:08:05 results of pre-diagnostic : light ok 42301, not duplicated 18335, two criteria ok 18335, nb forced upload 443 end of day status of photos as found in sqllite duplicated : 23958 nothing_done : 6 uploaded : 18759 ok_for_neural_classification : 6 Info from dashboard total number of uploaded images : 19144 number of useful images : 18790 number of finished photos : 15374 (81.82% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 3416 (18.18% of two criteria ok) number of finished batches : 254 (83.28% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 51 (16.72% of total) last update : 2025-03-28 06:19:03 You can launch datou with following url to update dashboard https://www.fotonower.com/api/v1/datou/current/set?mtr_portfolio_id=21570233,21715706,21758531&mtr_datou_id=4878 cvs resume : date,nb_photos,% time,nb ok,uploaded,to upload, % uploaded, nb useful, % completed photos, last_update, remark 2025/03/21,8527,97.37%,4148,4226,0,100.00%,4440,40.56%,2025/03/25, 2025/03/22,1777,97.79%,15,26,0,100.00%,0,0.00%,2025/03/23, 2025/03/24,6767,97.75%,3380,3451,0,100.00%,3415,100.00%,2025/03/25, 2025/03/25,8550,97.74%,2602,2700,0,100.00%,2632,100.00%,2025/03/26, 2025/03/26,8545,97.79%,3905,3998,6,99.85%,3967,82.20%,2025/03/27, 2025/03/27,8563,98.00%,4285,4358,0,100.00%,4336,98.36%,2025/03/28, coverage for this period for every 10 second 2025/03/21/ 0.9736686767207954 2025/03/22/ 0.977943219067726 2025/03/24/ 0.9774558107574672 2025/03/25/ 0.9773839974999711 2025/03/26/ 0.9778585400293985 2025/03/27/ 0.9772936661157331 mean value for this period : 0.9769339850318485 coverage for this period for every 20 second 2025/03/21/ 0.9897452516811537 2025/03/22/ 0.9899438857714318 2025/03/24/ 0.9920466673489379 2025/03/25/ 0.9923725969050567 2025/03/26/ 0.991793886503316 2025/03/27/ 0.9918702574572795 mean value for this period : 0.9912954242778623 3.01user 0.93system 0:25.64elapsed 15%CPU (0avgtext+0avgdata 194104maxresident)k 117936inputs+72outputs (32major+119114minor)pagefaults 0swaps