Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! DataTypes for each output/input checked ! Unexpected error treated as WARNING for now expected given param in final step {'min_score': 0.7, 'name_pipeline': 'proportion_papier_tapis', 'hashtag_type': 705, 'hashtag_proportion': 492668766, 'hashtag_parmi': 538914404, 'hashtag_to_consider': 'papier_only', 'hashtag_background': 493012381} Found hashtag_parmi in final step, will be used and useful if different from matiere_majoritaire in classifier ventilate_hashtags_in_portfolio step not found ! # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! Unexpected error treated as WARNING for now expected given param in final step {'name_pipeline': 'aggregate_carac_ratio', 'hashtag_type': 4230, 'hashtag_proportion': 'papier,carton,metal,pet_clair,autre,pehd,pet_fonce', 'hashtag_parmi': 'refus', 'hashtag_weights': {'autre': 3.0, 'barquette_avec_film': 3.0, 'carton': 3.0, 'metal': 3.0, 'papier': 3.0, 'pehd': 3.0, 'pet_fonce': 3.0, 'pet_clair': 1.0, 'flou': 1.0, 'mal_croppe': 1.0}, 'option_sccu': 1} Found hashtag_parmi in final step, will be used and useful if different from matiere_majoritaire in classifier Fonction SLA_RUBBIA TODO : Insert select and so on ***** analysis of device with port 20084 for dashboard dechet_Valcor_v2 ***** **** analysis of day 2025/04/24/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! **** analysis of day 2025/04/25/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 8514 coverage for 10 seconds : 23:05:18 (96.20%) coverage for 20 seconds : 23:43:11 (98.83%) max time between two photos : 0:01:20 results of pre-diagnostic : light ok 8532, not duplicated 7229, two criteria ok 7229, nb forced upload 0 end of day status of photos as found in sqllite duplicated : 1299 nothing_done : 1 ok_for_neural_classification : 1904 uploaded : 5310 Info from dashboard total number of uploaded images : 6366 number of useful images : 7229 number of finished photos : 1172 (16.21% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 318 (4.40% of two criteria ok) number of finished batches : 124 (70.45% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 52 (29.55% of total) last update : 2025-04-29 11:19:07 **** analysis of day 2025/04/26/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 8519 coverage for 10 seconds : 23:04:19 (96.13%) coverage for 20 seconds : 23:43:55 (98.88%) max time between two photos : 0:01:20 results of pre-diagnostic : light ok 8519, not duplicated 7474, two criteria ok 7474, nb forced upload 0 end of day status of photos as found in sqllite duplicated : 1045 nothing_done : 13 ok_for_neural_classification : 7461 Unable to find info for dashboard number 132 for day 2025-04-26 06:36:27.626047 Info from dashboard total number of uploaded images : 0 number of useful images : 7474 number of finished photos : 0 (0.00% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 0 (0.00% of two criteria ok) **** analysis of day 2025/04/27/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 8520 coverage for 10 seconds : 23:07:40 (96.37%) coverage for 20 seconds : 23:44:10 (98.90%) max time between two photos : 0:01:21 results of pre-diagnostic : light ok 8532, not duplicated 6681, two criteria ok 6681, nb forced upload 0 end of day status of photos as found in sqllite duplicated : 1848 nothing_done : 7 ok_for_neural_classification : 6665 Unable to find info for dashboard number 132 for day 2025-04-27 06:36:27.626047 Info from dashboard total number of uploaded images : 0 number of useful images : 6681 number of finished photos : 0 (0.00% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 0 (0.00% of two criteria ok) **** analysis of day 2025/04/28/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 8543 coverage for 10 seconds : 23:10:34 (96.57%) coverage for 20 seconds : 23:48:00 (99.17%) max time between two photos : 0:01:20 results of pre-diagnostic : light ok 8549, not duplicated 7009, two criteria ok 7009, nb forced upload 0 end of day status of photos as found in sqllite duplicated : 1537 nothing_done : 7 ok_for_neural_classification : 2523 uploaded : 4476 Info from dashboard total number of uploaded images : 4554 number of useful images : 7009 number of finished photos : 776 (11.07% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 0 (0.00% of two criteria ok) number of finished batches : 99 (100.00% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 0 (0.00% of total) last update : 2025-04-29 06:01:49 **** analysis of day 2025/04/29/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 8526 coverage for 10 seconds : 23:03:56 (96.11%) coverage for 20 seconds : 23:44:18 (98.91%) max time between two photos : 0:01:21 results of pre-diagnostic : light ok 8538, not duplicated 7389, two criteria ok 7389, nb forced upload 0 end of day status of photos as found in sqllite duplicated : 1138 nothing_done : 1 ok_for_neural_classification : 307 uploaded : 7080 Info from dashboard total number of uploaded images : 7200 number of useful images : 7389 number of finished photos : 2189 (29.63% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 25 (0.34% of two criteria ok) number of finished batches : 275 (99.64% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 1 (0.36% of total) last update : 2025-04-30 06:24:52 **** analysis of day 2025/04/30/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 8502 coverage for 10 seconds : 22:58:57 (95.76%) coverage for 20 seconds : 23:41:09 (98.69%) max time between two photos : 0:01:21 results of pre-diagnostic : light ok 8508, not duplicated 7610, two criteria ok 7610, nb forced upload 0 end of day status of photos as found in sqllite duplicated : 896 nothing_done : 7 ok_for_neural_classification : 543 uploaded : 7056 Info from dashboard total number of uploaded images : 7182 number of useful images : 7610 number of finished photos : 2019 (26.53% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 0 (0.00% of two criteria ok) number of finished batches : 256 (100.00% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 0 (0.00% of total) last update : 2025-05-01 06:01:29 **** analysis for all days **** Info from logs total number of images : 51124 coverage for 10 seconds : 5 days, 18:30:44 (96.19%) coverage for 20 seconds : 5 days, 22:24:43 (98.90%) max time between two photos : 0:08:03 results of pre-diagnostic : light ok 51178, not duplicated 43392, two criteria ok 43392, nb forced upload 0 end of day status of photos as found in sqllite duplicated : 7763 nothing_done : 36 ok_for_neural_classification : 19403 uploaded : 23922 Info from dashboard total number of uploaded images : 25302 number of useful images : 43392 number of finished photos : 6156 (14.19% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 343 (0.79% of two criteria ok) number of finished batches : 754 (93.43% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 53 (6.57% of total) last update : 2025-05-01 06:01:29 You can launch datou with following url to update dashboard https://www.fotonower.com/api/v1/datou/current/set?mtr_portfolio_id=22415669,22483834&mtr_datou_id=4311 cvs resume : date,nb_photos,% time,nb ok,uploaded,to upload, % uploaded, nb useful, % completed photos, last_update, remark 2025/04/25,8514,96.20%,7229,5310,1904,73.61%,1490,78.66%,2025/04/29, 2025/04/26,8519,96.13%,7474,0,7461,0.00%,0,0.00%,0000/00/00, 2025/04/27,8520,96.37%,6681,0,6665,0.00%,0,0.00%,0000/00/00, 2025/04/28,8543,96.57%,7009,4476,2523,63.95%,776,100.00%,2025/04/29, 2025/04/29,8526,96.11%,7389,7080,307,95.84%,2214,98.87%,2025/04/30, 2025/04/30,8502,95.76%,7610,7056,543,92.85%,2019,100.00%,2025/05/01, coverage for this period for every 10 second 2025/04/25/ 0.9620250234377713 2025/04/26/ 0.9613421451637172 2025/04/27/ 0.963668560978715 2025/04/28/ 0.9656824731767729 2025/04/29/ 0.9610759383789165 2025/04/30/ 0.9619015652187333 mean value for this period : 0.9626159510591044 coverage for this period for every 20 second 2025/04/25/ 0.9883331983009063 2025/04/26/ 0.9888424634544382 2025/04/27/ 0.9890160765749604 2025/04/28/ 0.9916781444229679 2025/04/29/ 0.9891086702392389 2025/04/30/ 0.9889832829855284 mean value for this period : 0.9893269726630067 2.62user 1.12system 0:15.19elapsed 24%CPU (0avgtext+0avgdata 198356maxresident)k 356080inputs+96outputs (129major+113194minor)pagefaults 0swaps