Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! DataTypes for each output/input checked ! Unexpected error treated as WARNING for now expected given param in final step {'min_score': 0.7, 'name_pipeline': 'proportion_papier_tapis', 'hashtag_type': 705, 'hashtag_proportion': 492668766, 'hashtag_parmi': 538914404, 'hashtag_to_consider': 'papier_only', 'hashtag_background': 493012381} Found hashtag_parmi in final step, will be used and useful if different from matiere_majoritaire in classifier ventilate_hashtags_in_portfolio step not found ! # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! Unexpected error treated as WARNING for now expected given param in final step {'name_pipeline': 'aggregate_carac_ratio', 'hashtag_type': 4230, 'hashtag_proportion': 'papier,carton,metal,pet_clair,autre,pehd,pet_fonce', 'hashtag_parmi': 'refus', 'hashtag_weights': {'autre': 3.0, 'barquette_avec_film': 3.0, 'carton': 3.0, 'metal': 3.0, 'papier': 3.0, 'pehd': 3.0, 'pet_fonce': 3.0, 'pet_clair': 1.0, 'flou': 1.0, 'mal_croppe': 1.0}, 'option_sccu': 1} Found hashtag_parmi in final step, will be used and useful if different from matiere_majoritaire in classifier Fonction SLA_RUBBIA TODO : Insert select and so on ***** analysis of device with port 20084 for dashboard dechet_Valcor_v2 ***** **** analysis of day 2025/05/21/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 8525 coverage for 10 seconds : 23:07:13 (96.34%) coverage for 20 seconds : 23:44:50 (98.95%) max time between two photos : 0:01:20 results of pre-diagnostic : light ok 8530, not duplicated 7298, two criteria ok 7298, nb forced upload 0 end of day status of photos as found in sqllite no_longer_on_disk : 1 duplicated : 1232 nothing_done : 7 ok_for_neural_classification : 217 uploaded : 7068 Info from dashboard total number of uploaded images : 7418 number of useful images : 7298 number of finished photos : 526 (7.21% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 0 (0.00% of two criteria ok) number of finished batches : 74 (100.00% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 0 (0.00% of total) last update : 2025-05-23 11:17:15 **** analysis of day 2025/05/22/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 8556 coverage for 10 seconds : 23:09:44 (96.51%) coverage for 20 seconds : 23:50:05 (99.31%) max time between two photos : 0:01:20 results of pre-diagnostic : light ok 8562, not duplicated 7390, two criteria ok 7390, nb forced upload 0 end of day status of photos as found in sqllite duplicated : 1172 nothing_done : 7 ok_for_neural_classification : 1581 uploaded : 5796 Info from dashboard total number of uploaded images : 7516 number of useful images : 7390 number of finished photos : 812 (10.99% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 0 (0.00% of two criteria ok) number of finished batches : 85 (100.00% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 0 (0.00% of total) last update : 2025-05-24 03:17:11 **** analysis of day 2025/05/23/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 8526 coverage for 10 seconds : 23:07:33 (96.36%) coverage for 20 seconds : 23:45:20 (98.98%) max time between two photos : 0:01:20 results of pre-diagnostic : light ok 8532, not duplicated 7259, two criteria ok 7259, nb forced upload 0 end of day status of photos as found in sqllite duplicated : 1270 nothing_done : 13 ok_for_neural_classification : 1963 uploaded : 5280 Info from dashboard total number of uploaded images : 7380 number of useful images : 7259 number of finished photos : 61 (0.84% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 1375 (18.94% of two criteria ok) number of finished batches : 13 (6.81% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 178 (93.19% of total) last update : 2025-05-27 15:27:06 **** analysis of day 2025/05/24/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 8517 coverage for 10 seconds : 23:04:04 (96.12%) coverage for 20 seconds : 23:43:35 (98.86%) max time between two photos : 0:01:20 results of pre-diagnostic : light ok 8532, not duplicated 7866, two criteria ok 7866, nb forced upload 0 end of day status of photos as found in sqllite duplicated : 664 nothing_done : 4 ok_for_neural_classification : 1357 uploaded : 6492 Info from dashboard total number of uploaded images : 7982 number of useful images : 7866 number of finished photos : 0 (0.00% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 352 (4.47% of two criteria ok) number of finished batches : 0 (0.00% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 49 (100.00% of total) last update : 2025-05-27 09:32:36 **** analysis of day 2025/05/25/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Catched exception ! ERROR in database : database disk image is malformed sqllite_today : /data_2/data_log/log_raspi/2025/05/25/20084/sqlite.db ERROR in database : database disk image is malformed sqllite_today : /data_2/data_log/log_raspi/2025/05/25/20084/sqlite.db Info from logs total number of images : 0 coverage for 10 seconds : 0:00:00 (0.00%) coverage for 20 seconds : 0:00:00 (0.00%) max time between two photos : 0:00:00 results of pre-diagnostic : light ok 8543, not duplicated 7740, two criteria ok 7739, nb forced upload 0 end of day status of photos as found in sqllite Info from dashboard total number of uploaded images : 7847 number of useful images : 7739 number of finished photos : 202 (2.61% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 6 (0.08% of two criteria ok) number of finished batches : 44 (97.78% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 1 (2.22% of total) last update : 2025-05-27 08:40:12 **** analysis of day 2025/05/26/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Catched exception ! ERROR in database : database disk image is malformed sqllite_today : /data_2/data_log/log_raspi/2025/05/26/20084/sqlite.db ERROR in database : database disk image is malformed sqllite_today : /data_2/data_log/log_raspi/2025/05/26/20084/sqlite.db Info from logs total number of images : 0 coverage for 10 seconds : 0:00:00 (0.00%) coverage for 20 seconds : 0:00:00 (0.00%) max time between two photos : 0:00:00 results of pre-diagnostic : light ok 8532, not duplicated 7765, two criteria ok 7765, nb forced upload 0 end of day status of photos as found in sqllite Info from dashboard total number of uploaded images : 7874 number of useful images : 7765 number of finished photos : 790 (10.17% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 315 (4.06% of two criteria ok) number of finished batches : 139 (79.89% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 35 (20.11% of total) last update : 2025-05-27 11:17:38 **** analysis of day 2025/05/27/ **** VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! VR TODO TOCHECK : due to this bug it shouldn't have being able to work, or maybe it was due to a change directory not done ! 0:00:00 apple pause between two photos 10 Find filename in sqlite. Info from logs total number of images : 8514 coverage for 10 seconds : 23:07:03 (96.32%) coverage for 20 seconds : 23:43:08 (98.83%) max time between two photos : 0:01:20 results of pre-diagnostic : light ok 8526, not duplicated 7963, two criteria ok 7963, nb forced upload 0 end of day status of photos as found in sqllite duplicated : 561 nothing_done : 7 ok_for_neural_classification : 950 uploaded : 6996 Info from dashboard total number of uploaded images : 8107 number of useful images : 7963 number of finished photos : 788 (9.90% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 17 (0.21% of two criteria ok) number of finished batches : 122 (96.83% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 4 (3.17% of total) last update : 2025-05-28 06:01:28 **** analysis for all days **** Info from logs total number of images : 42638 coverage for 10 seconds : 4 days, 19:35:37 (68.81%) coverage for 20 seconds : 4 days, 22:46:58 (70.70%) max time between two photos : 0:06:40 results of pre-diagnostic : light ok 59757, not duplicated 53281, two criteria ok 53280, nb forced upload 0 end of day status of photos as found in sqllite no_longer_on_disk : 1 duplicated : 4899 nothing_done : 38 ok_for_neural_classification : 6068 uploaded : 31632 Info from dashboard total number of uploaded images : 54124 number of useful images : 53280 number of finished photos : 3179 (5.97% of two criteria ok) number of finished photos for older algorithm : 0 (0.00% of two criteria ok) number of started photos : 2065 (3.88% of two criteria ok) number of finished batches : 477 (64.11% of total) number of finished batches for older algorithm : 0 (0.00% of total) number of started batches : 267 (35.89% of total) last update : 2025-05-28 06:01:28 You can launch datou with following url to update dashboard https://www.fotonower.com/api/v1/datou/current/set?mtr_portfolio_id=23285996,23312670,23315502,23319659,23363907&mtr_datou_id=4311 cvs resume : date,nb_photos,% time,nb ok,uploaded,to upload, % uploaded, nb useful, % completed photos, last_update, remark 2025/05/21,8525,96.34%,7298,7068,217,97.02%,526,100.00%,2025/05/23, 2025/05/22,8556,96.51%,7390,5796,1581,78.57%,812,100.00%,2025/05/24, 2025/05/23,8526,96.36%,7259,5280,1963,72.90%,1436,4.25%,2025/05/27, 2025/05/24,8517,96.12%,7866,6492,1357,82.71%,352,0.00%,2025/05/27, 2025/05/25,0,0.00%,7739,0,0,0.00%,208,97.12%,2025/05/27, 2025/05/26,0,0.00%,7765,0,0,0.00%,1105,71.49%,2025/05/27, 2025/05/27,8514,96.32%,7963,6996,950,88.04%,805,97.89%,2025/05/28, coverage for this period for every 10 second 2025/05/21/ 0.963356057361775 2025/05/22/ 0.9651037627750321 2025/05/23/ 0.9635875415224713 2025/05/24/ 0.961168532043195 2025/05/25/ 0.0 2025/05/26/ 0.0 2025/05/27/ 0.6880651727119858 mean value for this period : 0.6487544380592085 coverage for this period for every 20 second 2025/05/21/ 0.989479044896353 2025/05/22/ 0.9931249204273198 2025/05/23/ 0.9898262711373974 2025/05/24/ 0.9886109792937419 2025/05/25/ 0.0 2025/05/26/ 0.0 2025/05/27/ 0.7070485273473734 mean value for this period : 0.6668699633003122 3.00user 1.08system 0:23.82elapsed 17%CPU (0avgtext+0avgdata 196000maxresident)k 399656inputs+96outputs (15major+94491minor)pagefaults 0swaps