python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2701424 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 20>10 please execute split_portfolio.py -i 20905796 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2621865'] with mtr_portfolio_ids : ['20905796'] and first list_photo_ids : [] new path : /proc/2701424/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 114$œ’m…Îmo“LÒDüÜ;˜%gÏ?wêÁÅ·øîùovH0õÉa‡5£Ú*î Ø’ÃÌlÍ››S iyä”rÕO7ª“ž�%L]Ý×%±ºÇhk¶«·÷>v1­HB£®±ßÞÚd\�(e oIx¢>3´6BS%ÌØá“( œÛf$Ãhýé ¿¶åeÔŽôÚèHœ�‚`ݶ f{�Fo©Yò¿Ôó@00uMb’z-ëìXI$&ÂgfÖú¶7Ó´Þu|'K.ÌoP PÝÀùFË.Ðýoûò9B<~.’ïÅË[’´˜Ë<Ù­„$¯•¢·ä{1¹A�•.òbKxºL �¯�Ý·'¯u8n5 ’ºe,]ñH©–’ÆV¨ŒWwÃ$ùCƒel¹“|zys«™KŠi-ðqÊݬbk,wnGÿâ;¥ ~ ÖeÉrÍ’‰ÜÔ~' 1`V⦫¹-*[ÉñLÔKÄ'2@ŸÜþÐ仪 ²n‘Íß2¸N߈ƶµG•¢ói/U¢µ'Eï@¦`H湘;J•�¼¼ÜŽÅ +Jén#»¼‚6Ú´—ĹG•ü¡NÒGð'—Z!öáí¸‰Wi�»NJ @óàšAûÜZ|ª[¨ï$q}i�Ò·µQbtTEC$œ’m…Îmo“LÒDüÜ;˜%gÏ?wêÁÅ·øîùovH0õÉa‡5£Ú*î Ø’ÃÌlÍ››S iyä”rÕO7ª“ž�%L]Ý×%±ºÇhk¶«·÷>v1­HB£®±ßÞÚd\�(e oIx¢>3´6BS%ÌØá“( œÛf$Ãhýé ¿¶åeÔŽôÚèHœ�‚`ݶ f{�Fo©Yò¿Ôó@00uMb’z-ëìXI$&ÂgfÖú¶7Ó´Þu|'K.ÌoP PÝÀùFË.Ðýoûò9B<~.’ïÅË[’´˜Ë<Ù­„$¯•¢·ä{1¹A�•.òbKxºL �¯�Ý·'¯u8n5 ’ºe,]ñH©–’ÆV¨ŒWwÃ$ùCƒel¹“|zys«™KŠi-ðqÊݬbk,wnGÿâ;¥ ~ ÖeÉrÍ’‰ÜÔ~' 1`V⦫¹-*[ÉñLÔKÄ'2@ŸÜþÐ仪 ²n‘Íß2¸N߈ƶµG•¢ói/U¢µ'Eï@¦`H湘;J•�¼¼ÜŽÅ +Jén#»¼‚6Ú´—ĹG•ü¡NÒGð'—Z!öáí¸‰Wi�»NJ @óàšAûÜZ|ª[¨ï$q}i�Ò·µQbtTEC$œ’m…Îmo“LÒDüÜ;˜%gÏ?wêÁÅ·øîùovH0õÉa‡5£Ú*î Ø’ÃÌlÍ››S iyä”rÕO7ª“ž�%L]Ý×%±ºÇhk¶«·÷>v1­HB£®±ßÞÚd\�(e oIx¢>3´6BS%ÌØá“( œÛf$Ãhýé ¿¶åeÔŽôÚèHœ�‚`ݶ f{�Fo©Yò¿Ôó@00uMb’z-ëìXI$&ÂgfÖú¶7Ó´Þu|'K.ÌoP PÝÀùFË.Ðýoûò9B<~.’ïÅË[’´˜Ë<Ù­„$¯•¢·ä{1¹A�•.òbKxºL �¯�Ý·'¯u8n5 ’ºe,]ñH©–’ÆV¨ŒWwÃ$ùCƒel¹“|zys«™KŠi-ðqÊݬbk,wnGÿâ;¥ ~ ÖeÉrÍ’‰ÜÔ~' 1`V⦫¹-*[ÉñLÔKÄ'2@ŸÜþÐ仪 ²n‘Íß2¸N߈ƶµG•¢ói/U¢µ'Eï@¦`H湘;J•�¼¼ÜŽÅ +Jén#»¼‚6Ú´—ĹG•ü¡NÒGð'—Z!öáí¸‰Wi�»NJ @óàšAûÜZ|ª[¨ï$q}i�Ò·µQbtTEC$œ’m…Îmo“LÒDüÜ;˜%gÏ?wêÁÅ·øîùovH0õÉa‡5£Ú*î Ø’ÃÌlÍ››S iyä”rÕO7ª“ž�%L]Ý×%±ºÇhk¶«·÷>v1­HB£®±ßÞÚd\�(e oIx¢>3´6BS%ÌØá“( œÛf$Ãhýé ¿¶åeÔŽôÚèHœ�‚`ݶ f{�Fo©Yò¿Ôó@00uMb’z-ëìXI$&ÂgfÖú¶7Ó´Þu|'K.ÌoP PÝÀùFË.Ðýoûò9B<~.’ïÅË[’´˜Ë<Ù­„$¯•¢·ä{1¹A�•.òbKxºL �¯�Ý·'¯u8n5 ’ºe,]ñH©–’ÆV¨ŒWwÃ$ùCƒel¹“|zys«™KŠi-ðqÊݬbk,wnGÿâ;¥ ~ ÖeÉrÍ’‰ÜÔ~' 1`V⦫¹-*[ÉñLÔKÄ'2@ŸÜþÐ仪 ²n‘Íß2¸N߈ƶµG•¢ói/U¢µ'Eï@¦`H湘;J•�¼¼ÜŽÅ +Jén#»¼‚6Ú´—ĹG•ü¡NÒGð'—Z!öáí¸‰Wi�»NJ @óàšAûÜZ|ª[¨ï$q}i�Ò·µQbtTEC$œ’m…Îmo“LÒDüÜ;˜%gÏ?wêÁÅ·øîùovH0õÉa‡5£Ú*î Ø’ÃÌlÍ››S iyä”rÕO7ª“ž�%L]Ý×%±ºÇhk¶«·÷>v1­HB£®±ßÞÚd\�(e oIx¢>3´6BS%ÌØá“( œÛf$Ãhýé ¿¶åeÔŽôÚèHœ�‚`ݶ f{�Fo©Yò¿Ôó@00uMb’z-ëìXI$&ÂgfÖú¶7Ó´Þu|'K.ÌoP PÝÀùFË.Ðýoûò9B<~.’ïÅË[’´˜Ë<Ù­„$¯•¢·ä{1¹A�•.òbKxºL �¯�Ý·'¯u8n5 ’ºe,]ñH©–’ÆV¨ŒWwÃ$ùCƒel¹“|zys«™KŠi-ðqÊݬbk,wnGÿâ;¥ ~ ÖeÉrÍ’‰ÜÔ~' 1`V⦫¹-*[ÉñLÔKÄ'2@ŸÜþÐ仪 ²n‘Íß2¸N߈ƶµG•¢ói/U¢µ'Eï@¦`H湘;J•�¼¼ÜŽÅ +Jén#»¼‚6Ú´—ĹG•ü¡NÒGð'—Z!öáí¸‰Wi�»NJ @óàšAûÜZ|ª[¨ï$q}i�Ò·µQbtTEC$œ’m…Îmo“LÒDüÜ;˜%gÏ?wêÁÅ·øîùovH0õÉa‡5£Ú*î Ø’ÃÌlÍ››S iyä”rÕO7ª“ž�%L]Ý×%±ºÇhk¶«·÷>v1­HB£®±ßÞÚd\�(e oIx¢>3´6BS%ÌØá“( œÛf$Ãhýé ¿¶åeÔŽôÚèHœ�‚`ݶ f{�Fo©Yò¿Ôó@00uMb’z-ëìXI$&ÂgfÖú¶7Ó´Þu|'K.ÌoP PÝÀùFË.Ðýoûò9B<~.’ïÅË[’´˜Ë<Ù­„$¯•¢·ä{1¹A�•.òbKxºL �¯�Ý·'¯u8n5 ’ºe,]ñH©–’ÆV¨ŒWwÃ$ùCƒel¹“|zys«™KŠi-ðqÊݬbk,wnGÿâ;¥ ~ ÖeÉrÍ’‰ÜÔ~' 1`V⦫¹-*[ÉñLÔKÄ'2@ŸÜþÐ仪 ²n‘Íß2¸N߈ƶµG•¢ói/U¢µ'Eï@¦`H湘;J•�¼¼ÜŽÅ +Jén#»¼‚6Ú´—ĹG•ü¡NÒGð'—Z!öáí¸‰Wi�»NJ @óàšAûÜZ|ª[¨ï$q}i�Ò·µQbtTEC$œ’m…Îmo“LÒDüÜ;˜%gÏ?wêÁÅ·øîùovH0õÉa‡5£Ú*î Ø’ÃÌlÍ››S iyä”rÕO7ª“ž�%L]Ý×%±ºÇhk¶«·÷>v1­HB£®±ßÞÚd\�(e oIx¢>3´6BS%ÌØá“( œÛf$Ãhýé ¿¶åeÔŽôÚèHœ�‚`ݶ f{�Fo©Yò¿Ôó@00uMb’z-ëìXI$&ÂgfÖú¶7Ó´Þu|'K.ÌoP PÝÀùFË.Ðýoûò9B<~.’ïÅË[’´˜Ë<Ù­„$¯•¢·ä{1¹A�•.òbKxºL �¯�Ý·'¯u8n5 ’ºe,]ñH©–’ÆV¨ŒWwÃ$ùCƒel¹“|zys«™KŠi-ðqÊݬbk,wnGÿâ;¥ ~ ÖeÉrÍ’‰ÜÔ~' 1`V⦫¹-*[ÉñLÔKÄ'2@ŸÜþÐ仪 ²n‘Íß2¸N߈ƶµG•¢ói/U¢µ'Eï@¦`H湘;J•�¼¼ÜŽÅ +Jén#»¼‚6Ú´—ĹG•ü¡NÒGð'—Z!öáí¸‰Wi�»NJ @óàšAûÜZ|ª[¨ï$q}i�Ò·µQbtTEC$œ’m…Îmo“LÒDüÜ;˜%gÏ?wêÁÅ·øîùovH0õÉa‡5£Ú*î Ø’ÃÌlÍ››S iyä”rÕO7ª“ž�%L]Ý×%±ºÇhk¶«·÷>v1­HB£®±ßÞÚd\�(e oIx¢>3´6BS%ÌØá“( œÛf$Ãhýé ¿¶åeÔŽôÚèHœ�‚`ݶ f{�Fo©Yò¿Ôó@00uMb’z-ëìXI$&ÂgfÖú¶7Ó´Þu|'K.ÌoP PÝÀùFË.Ðýoûò9B<~.’ïÅË[’´˜Ë<Ù­„$¯•¢·ä{1¹A�•.òbKxºL �¯�Ý·'¯u8n5 ’ºe,]ñH©–’ÆV¨ŒWwÃ$ùCƒel¹“|zys«™KŠi-ðqÊݬbk,wnGÿâ;¥ ~ ÖeÉrÍ’‰ÜÔ~' 1`V⦫¹-*[ÉñLÔKÄ'2@ŸÜþÐ仪 ²n‘Íß2¸N߈ƶµG•¢ói/U¢µ'Eï@¦`H湘;J•�¼¼ÜŽÅ +Jén#»¼‚6Ú´—ĹG•ü¡NÒGð'—Z!öáí¸‰Wi�»NJ @óàšAûÜZ|ª[¨ï$q}i�Ò·µQbtTEC$œ’m…Îmo“LÒDüÜ;˜%gÏ?wêÁÅ·øîùovH0õÉa‡5£Ú*î Ø’ÃÌlÍ››S iyä”rÕO7ª“ž�%L]Ý×%±ºÇhk¶«·÷>v1­HB£®±ßÞÚd\�(e oIx¢>3´6BS%ÌØá“( œÛf$Ãhýé ¿¶åeÔŽôÚèHœ�‚`ݶ f{�Fo©Yò¿Ôó@00uMb’z-ëìXI$&ÂgfÖú¶7Ó´Þu|'K.ÌoP PÝÀùFË.Ðýoûò9B<~.’ïÅË[’´˜Ë<Ù­„$¯•¢·ä{1¹A�•.òbKxºL �¯�Ý·'¯u8n5 ’ºe,]ñH©–’ÆV¨ŒWwÃ$ùCƒel¹“|zys«™KŠi-ðqÊݬbk,wnGÿâ;¥ ~ ÖeÉrÍ’‰ÜÔ~' 1`V⦫¹-*[ÉñLÔKÄ'2@ŸÜþÐ仪 ²n‘Íß2¸N߈ƶµG•¢ói/U¢µ'Eï@¦`H湘;J•�¼¼ÜŽÅ +Jén#»¼‚6Ú´—ĹG•ü¡NÒGð'—Z!öáí¸‰Wi�»NJ @óàšAûÜZ|ª[¨ï$q}i�Ò·µQbtTEC$œ’m…Îmo“LÒDüÜ;˜%gÏ?wêÁÅ·øîùovH0õÉa‡5£Ú*î Ø’ÃÌlÍ››S iyä”rÕO7ª“ž�%L]Ý×%±ºÇhk¶«·÷>v1­HB£®±ßÞÚd\�(e oIx¢>3´6BS%ÌØá“( œÛf$Ãhýé ¿¶åeÔŽôÚèHœ�‚`ݶ f{�Fo©Yò¿Ôó@00uMb’z-ëìXI$&ÂgfÖú¶7Ó´Þu|'K.ÌoP PÝÀùFË.Ðýoûò9B<~.’ïÅË[’´˜Ë<Ù­„$¯•¢·ä{1¹A�•.òbKxºL �¯�Ý·'¯u8n5 ’ºe,]ñH©–’ÆV¨ŒWwÃ$ùCƒel¹“|zys«™KŠi-ðqÊݬbk,wnGÿâ;¥ ~ ÖeÉrÍ’‰ÜÔ~' 1`V⦫¹-*[ÉñLÔKÄ'2@ŸÜþÐ仪 ²n‘Íß2¸N߈ƶµG•¢ói/U¢µ'Eï@¦`H湘;J•�¼¼ÜŽÅ +Jén#»¼‚6Ú´—ĹG•ü¡NÒGð'—Z!öáí¸‰Wi�»NJ @óàšAûÜZ|ª[¨ï$q}i�Ò·µQbtTEC$œ’m…Îmo“LÒDüÜ;˜%gÏ?wêÁÅ·øîùovH0õÉa‡5£Ú*î Ø’ÃÌlÍ››S iyä”rÕO7ª“ž�%L]Ý×%±ºÇhk¶«·÷>v1­HB£®±ßÞÚd\�(e oIx¢>3´6BS%ÌØá“( œÛf$Ãhýé ¿¶åeÔŽôÚèHœ�‚`ݶ f{�Fo©Yò¿Ôó@00uMb’z-ëìXI$&ÂgfÖú¶7Ó´Þu|'K.ÌoP PÝÀùFË.Ðýoûò9B<~.’ïÅË[’´˜Ë<Ù­„$¯•¢·ä{1¹A�•.òbKxºL �¯�Ý·'¯u8n5 ’ºe,]ñH©–’ÆV¨ŒWwÃ$ùCƒel¹“|zys«™KŠi-ðqÊݬbk,wnGÿâ;¥ ~ ÖeÉrÍ’‰ÜÔ~' 1`V⦫¹-*[ÉñLÔKÄ'2@ŸÜþÐ仪 ²n‘Íß2¸N߈ƶµG•¢ói/U¢µ'Eï@¦`H湘;J•�¼¼ÜŽÅ +Jén#»¼‚6Ú´—ĹG•ü¡NÒGð'—Z!öáí¸‰Wi�»NJ @óàšAûÜZ|ª[¨ï$q}i�Ò·µQbtTEC$œ’m…Îmo“LÒDüÜ;˜%gÏ?wêÁÅ·øîùovH0õÉa‡5£Ú*î Ø’ÃÌlÍ››S iyä”rÕO7ª“ž�%L]Ý×%±ºÇhk¶«·÷>v1­HB£®±ßÞÚd\�(e oIx¢>3´6BS%ÌØá“( œÛf$Ãhýé ¿¶åeÔŽôÚèHœ�‚`ݶ f{�Fo©Yò¿Ôó@00uMb’z-ëìXI$&ÂgfÖú¶7Ó´Þu|'K.ÌoP PÝÀùFË.Ðýoûò9B<~.’ïÅË[’´˜Ë<Ù­„$¯•¢·ä{1¹A�•.òbKxºL �¯�Ý·'¯u8n5 ’ºe,]ñH©–’ÆV¨ŒWwÃ$ùCƒel¹“|zys«™KŠi-ðqÊݬbk,wnGÿâ;¥ ~ ÖeÉrÍ’‰ÜÔ~' 1`V⦫¹-*[ÉñLÔKÄ'2@ŸÜþÐ仪 ²n‘Íß2¸N߈ƶµG•¢ói/U¢µ'Eï@¦`H湘;J•�¼¼ÜŽÅ +Jén#»¼‚6Ú´—ĹG•ü¡NÒGð'—Z!öáí¸‰Wi�»NJ @óàšAûÜZ|ª[¨ï$q}i�Ò·µQbtTEC$œ’m…Îmo“LÒDüÜ;˜%gÏ?wêÁÅ·øîùovH0õÉa‡5£Ú*î Ø’ÃÌlÍ››S iyä”rÕO7ª“ž�%L]Ý×%±ºÇhk¶«·÷>v1­HB£®±ßÞÚd\�(e oIx¢>3´6BS%ÌØá“( œÛf$Ãhýé ¿¶åeÔŽôÚèHœ�‚`ݶ f{�Fo©Yò¿Ôó@00uMb’z-ëìXI$&ÂgfÖú¶7Ó´Þu|'K.ÌoP PÝÀùFË.Ðýoûò9B<~.’ïÅË[’´˜Ë<Ù­„$¯•¢·ä{1¹A�•.òbKxºL �¯�Ý·'¯u8n5 ’ºe,]ñH©–’ÆV¨ŒWwÃ$ùCƒel¹“|zys«™KŠi-ðqÊݬbk,wnGÿâ;¥ ~ ÖeÉrÍ’‰ÜÔ~' 1`V⦫¹-*[ÉñLÔKÄ'2@ŸÜþÐ仪 ²n‘Íß2¸N߈ƶµG•¢ói/U¢µ'Eï@¦`H湘;J•�¼¼ÜŽÅ +Jén#»¼‚6Ú´—ĹG•ü¡NÒGð'—Z!öáí¸‰Wi�»NJ @óàšAûÜZ|ª[¨ï$q}i�Ò·µQbtTEC$œ’m…Îmo“LÒDüÜ;˜%gÏ?wêÁÅ·øîùovH0õÉa‡5£Ú*î Ø’ÃÌlÍ››S iyä”rÕO7ª“ž�%L]Ý×%±ºÇhk¶«·÷>v1­HB£®±ßÞÚd\�(e oIx¢>3´6BS%ÌØá“( œÛf$Ãhýé ¿¶åeÔŽôÚèHœ�‚`ݶ f{�Fo©Yò¿Ôó@00uMb’z-ëìXI$&ÂgfÖú¶7Ó´Þu|'K.ÌoP PÝÀùFË.Ðýoûò9B<~.’ïÅË[’´˜Ë<Ù­„$¯•¢·ä{1¹A�•.òbKxºL �¯�Ý·'¯u8n5 ’ºe,]ñH©–’ÆV¨ŒWwÃ$ùCƒel¹“|zys«™KŠi-ðqÊݬbk,wnGÿâ;¥ ~ ÖeÉrÍ’‰ÜÔ~' 1`V⦫¹-*[ÉñLÔKÄ'2@ŸÜþÐ仪 ²n‘Íß2¸N߈ƶµG•¢ói/U¢µ'Eï@¦`H湘;J•�¼¼ÜŽÅ +Jén#»¼‚6Ú´—ĹG•ü¡NÒGð'—Z!öáí¸‰Wi�»NJ @óàšAûÜZ|ª[¨ï$q}i�Ò·µQbtTEC$œ’m…Îmo“LÒDüÜ;˜%gÏ?wêÁÅ·øîùovH0õÉa‡5£Ú*î Ø’ÃÌlÍ››S iyä”rÕO7ª“ž�%L]Ý×%±ºÇhk¶«·÷>v1­HB£®±ßÞÚd\�(e oIx¢>3´6BS%ÌØá“( œÛf$Ãhýé ¿¶åeÔŽôÚèHœ�‚`ݶ f{�Fo©Yò¿Ôó@00uMb’z-ëìXI$&ÂgfÖú¶7Ó´Þu|'K.ÌoP PÝÀùFË.Ðýoûò9B<~.’ïÅË[’´˜Ë<Ù­„$¯•¢·ä{1¹A�•.òbKxºL �¯�Ý·'¯u8n5 ’ºe,]ñH©–’ÆV¨ŒWwÃ$ùCƒel¹“|zys«™KŠi-ðqÊݬbk,wnGÿâ;¥ ~ ÖeÉrÍ’‰ÜÔ~' 1`V⦫¹-*[ÉñLÔKÄ'2@ŸÜþÐ仪 ²n‘Íß2¸N߈ƶµG•¢ói/U¢µ'Eï@¦`H湘;J•�¼¼ÜŽÅ +Jén#»¼‚6Ú´—ĹG•ü¡NÒGð'—Z!öáí¸‰Wi�»NJ @óàšAûÜZ|ª[¨ï$q}i�Ò·µQbtTEC$œ’m…Îmo“LÒDüÜ;˜%gÏ?wêÁÅ·øîùovH0õÉa‡5£Ú*î Ø’ÃÌlÍ››S iyä”rÕO7ª“ž�%L]Ý×%±ºÇhk¶«·÷>v1­HB£®±ßÞÚd\�(e oIx¢>3´6BS%ÌØá“( œÛf$Ãhýé ¿¶åeÔŽôÚèHœ�‚`ݶ f{�Fo©Yò¿Ôó@00uMb’z-ëìXI$&ÂgfÖú¶7Ó´Þu|'K.ÌoP PÝÀùFË.Ðýoûò9B<~.’ïÅË[’´˜Ë<Ù­„$¯•¢·ä{1¹A�•.òbKxºL �¯�Ý·'¯u8n5 ’ºe,]ñH©–’ÆV¨ŒWwÃ$ùCƒel¹“|zys«™KŠi-ðqÊݬbk,wnGÿâ;¥ ~ ÖeÉrÍ’‰ÜÔ~' 1`V⦫¹-*[ÉñLÔKÄ'2@ŸÜþÐ仪 ²n‘Íß2¸N߈ƶµG•¢ói/U¢µ'Eï@¦`H湘;J•�¼¼ÜŽÅ +Jén#»¼‚6Ú´—ĹG•ü¡NÒGð'—Z!öáí¸‰Wi�»NJ @óàšAûÜZ|ª[¨ï$q}i�Ò·µQbtTECce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-28 03:18:34.469002: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-28 03:18:34.469048: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-28 03:18:34.469071: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-28 03:18:34.469089: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-28 03:18:34.469106: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-28 03:18:34.469122: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-28 03:18:34.469138: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-28 03:18:34.470082: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-28 03:18:34.471345: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-28 03:18:34.471406: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-28 03:18:34.471422: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-28 03:18:34.471436: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-28 03:18:34.471450: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-28 03:18:34.471464: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-28 03:18:34.471478: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-28 03:18:34.471492: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-28 03:18:34.472450: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-28 03:18:34.472490: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-28 03:18:34.472499: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-28 03:18:34.472507: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-28 03:18:34.473524: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4410 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-28 03:18:45.744881: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-28 03:18:45.962650: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-28 03:18:47.446737: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.81G (4089315328 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 15 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 25 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 1 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 150.61604 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 16 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 19 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 16 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 23 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 2 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 15 Detection mask done ! Trying to reset tf kernel 2701966 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 159 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 4862 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.001081705093383789 nb_pixel_total : 52918 time to create 1 rle with old method : 0.062161922454833984 length of segment : 301 time for calcul the mask position with numpy : 5.91278076171875e-05 nb_pixel_total : 981 time to create 1 rle with old method : 0.0012760162353515625 length of segment : 25 time for calcul the mask position with numpy : 0.0001811981201171875 nb_pixel_total : 8110 time to create 1 rle with old method : 0.00997614860534668 length of segment : 65 time for calcul the mask position with numpy : 0.00013208389282226562 nb_pixel_total : 6531 time to create 1 rle with old method : 0.007856130599975586 length of segment : 108 time for calcul the mask position with numpy : 0.00023245811462402344 nb_pixel_total : 15086 time to create 1 rle with old method : 0.01790618896484375 length of segment : 106 time for calcul the mask position with numpy : 0.0002484321594238281 nb_pixel_total : 12855 time to create 1 rle with old method : 0.01482081413269043 length of segment : 187 time for calcul the mask position with numpy : 0.0001270771026611328 nb_pixel_total : 6330 time to create 1 rle with old method : 0.007767438888549805 length of segment : 73 time for calcul the mask position with numpy : 0.000263214111328125 nb_pixel_total : 15318 time to create 1 rle with old method : 0.018421173095703125 length of segment : 169 time for calcul the mask position with numpy : 0.003139019012451172 nb_pixel_total : 208338 time to create 1 rle with new method : 0.009716272354125977 length of segment : 560 time for calcul the mask position with numpy : 0.001096487045288086 nb_pixel_total : 78979 time to create 1 rle with old method : 0.09181642532348633 length of segment : 253 time for calcul the mask position with numpy : 0.0002105236053466797 nb_pixel_total : 12992 time to create 1 rle with old method : 0.01531839370727539 length of segment : 122 time for calcul the mask position with numpy : 0.0012462139129638672 nb_pixel_total : 49506 time to create 1 rle with old method : 0.05964803695678711 length of segment : 446 time for calcul the mask position with numpy : 0.00350189208984375 nb_pixel_total : 282370 time to create 1 rle with new method : 0.008914470672607422 length of segment : 492 time for calcul the mask position with numpy : 0.0011022090911865234 nb_pixel_total : 59601 time to create 1 rle with old method : 0.0778806209564209 length of segment : 505 time for calcul the mask position with numpy : 0.0002543926239013672 nb_pixel_total : 6980 time to create 1 rle with old method : 0.008577585220336914 length of segment : 137 time for calcul the mask position with numpy : 0.0007193088531494141 nb_pixel_total : 31177 time to create 1 rle with old method : 0.03526496887207031 length of segment : 428 time for calcul the mask position with numpy : 0.008588314056396484 nb_pixel_total : 357833 time to create 1 rle with new method : 0.052847862243652344 length of segment : 1389 time for calcul the mask position with numpy : 0.0005338191986083984 nb_pixel_total : 22314 time to create 1 rle with old method : 0.027039766311645508 length of segment : 203 time for calcul the mask position with numpy : 0.0002899169921875 nb_pixel_total : 10002 time to create 1 rle with old method : 0.011694192886352539 length of segment : 199 time for calcul the mask position with numpy : 0.0006449222564697266 nb_pixel_total : 23070 time to create 1 rle with old method : 0.02706766128540039 length of segment : 214 time for calcul the mask position with numpy : 7.772445678710938e-05 nb_pixel_total : 1427 time to create 1 rle with old method : 0.0017802715301513672 length of segment : 36 time for calcul the mask position with numpy : 0.0009608268737792969 nb_pixel_total : 54787 time to create 1 rle with old method : 0.062303781509399414 length of segment : 198 time for calcul the mask position with numpy : 0.002819061279296875 nb_pixel_total : 162423 time to create 1 rle with new method : 0.008420705795288086 length of segment : 877 time for calcul the mask position with numpy : 0.00014519691467285156 nb_pixel_total : 4179 time to create 1 rle with old method : 0.0050961971282958984 length of segment : 76 time for calcul the mask position with numpy : 0.001775979995727539 nb_pixel_total : 96915 time to create 1 rle with old method : 0.1148519515991211 length of segment : 395 time for calcul the mask position with numpy : 0.0023195743560791016 nb_pixel_total : 128275 time to create 1 rle with old method : 0.1710827350616455 length of segment : 445 time for calcul the mask position with numpy : 0.0013921260833740234 nb_pixel_total : 46656 time to create 1 rle with old method : 0.05349588394165039 length of segment : 423 time for calcul the mask position with numpy : 0.0013649463653564453 nb_pixel_total : 53766 time to create 1 rle with old method : 0.06386017799377441 length of segment : 429 time for calcul the mask position with numpy : 0.0001881122589111328 nb_pixel_total : 3121 time to create 1 rle with old method : 0.005180835723876953 length of segment : 70 time for calcul the mask position with numpy : 0.002626180648803711 nb_pixel_total : 150493 time to create 1 rle with new method : 0.008198261260986328 length of segment : 465 time for calcul the mask position with numpy : 0.0003151893615722656 nb_pixel_total : 11766 time to create 1 rle with old method : 0.01497793197631836 length of segment : 163 time for calcul the mask position with numpy : 0.557443380355835 nb_pixel_total : 3856236 time to create 1 rle with new method : 0.848261833190918 length of segment : 3391 time for calcul the mask position with numpy : 0.0004858970642089844 nb_pixel_total : 9195 time to create 1 rle with old method : 0.01115560531616211 length of segment : 131 time for calcul the mask position with numpy : 0.003113985061645508 nb_pixel_total : 99846 time to create 1 rle with old method : 0.1185765266418457 length of segment : 509 time for calcul the mask position with numpy : 0.0011234283447265625 nb_pixel_total : 39272 time to create 1 rle with old method : 0.04802227020263672 length of segment : 242 time for calcul the mask position with numpy : 0.0003402233123779297 nb_pixel_total : 10451 time to create 1 rle with old method : 0.012824296951293945 length of segment : 97 time for calcul the mask position with numpy : 0.01633763313293457 nb_pixel_total : 608369 time to create 1 rle with new method : 0.044701576232910156 length of segment : 1874 time for calcul the mask position with numpy : 0.0007386207580566406 nb_pixel_total : 26609 time to create 1 rle with old method : 0.032384634017944336 length of segment : 371 time for calcul the mask position with numpy : 0.00022864341735839844 nb_pixel_total : 2909 time to create 1 rle with old method : 0.005305051803588867 length of segment : 69 time for calcul the mask position with numpy : 0.001990079879760742 nb_pixel_total : 50251 time to create 1 rle with old method : 0.06551122665405273 length of segment : 586 time for calcul the mask position with numpy : 0.00032210350036621094 nb_pixel_total : 3654 time to create 1 rle with old method : 0.005158185958862305 length of segment : 156 time for calcul the mask position with numpy : 0.0010347366333007812 nb_pixel_total : 35596 time to create 1 rle with old method : 0.045975446701049805 length of segment : 176 time for calcul the mask position with numpy : 0.00017905235290527344 nb_pixel_total : 8155 time to create 1 rle with old method : 0.009877681732177734 length of segment : 162 time for calcul the mask position with numpy : 0.004125833511352539 nb_pixel_total : 191061 time to create 1 rle with new method : 0.009259462356567383 length of segment : 578 time for calcul the mask position with numpy : 0.0071713924407958984 nb_pixel_total : 237233 time to create 1 rle with new method : 0.014998435974121094 length of segment : 586 time for calcul the mask position with numpy : 0.00020122528076171875 nb_pixel_total : 3168 time to create 1 rle with old method : 0.004195690155029297 length of segment : 85 time for calcul the mask position with numpy : 0.05694174766540527 nb_pixel_total : 2152953 time to create 1 rle with new method : 0.5174150466918945 length of segment : 2343 time for calcul the mask position with numpy : 0.0001659393310546875 nb_pixel_total : 1464 time to create 1 rle with old method : 0.001963376998901367 length of segment : 56 time for calcul the mask position with numpy : 0.00081634521484375 nb_pixel_total : 67716 time to create 1 rle with old method : 0.08203482627868652 length of segment : 263 time for calcul the mask position with numpy : 0.003061532974243164 nb_pixel_total : 221046 time to create 1 rle with new method : 0.006916046142578125 length of segment : 646 time for calcul the mask position with numpy : 0.0023543834686279297 nb_pixel_total : 200742 time to create 1 rle with new method : 0.0047152042388916016 length of segment : 605 time for calcul the mask position with numpy : 0.007220029830932617 nb_pixel_total : 300595 time to create 1 rle with new method : 0.018435001373291016 length of segment : 1831 time for calcul the mask position with numpy : 0.002744436264038086 nb_pixel_total : 128438 time to create 1 rle with old method : 0.15476393699645996 length of segment : 535 time for calcul the mask position with numpy : 0.00015354156494140625 nb_pixel_total : 3833 time to create 1 rle with old method : 0.004860401153564453 length of segment : 81 time for calcul the mask position with numpy : 0.0004107952117919922 nb_pixel_total : 7016 time to create 1 rle with old method : 0.008760929107666016 length of segment : 203 time for calcul the mask position with numpy : 0.0007863044738769531 nb_pixel_total : 31980 time to create 1 rle with old method : 0.0396575927734375 length of segment : 290 time for calcul the mask position with numpy : 0.0023436546325683594 nb_pixel_total : 114632 time to create 1 rle with old method : 0.14247679710388184 length of segment : 511 time for calcul the mask position with numpy : 0.00023627281188964844 nb_pixel_total : 5137 time to create 1 rle with old method : 0.006588459014892578 length of segment : 84 time for calcul the mask position with numpy : 0.0005908012390136719 nb_pixel_total : 26631 time to create 1 rle with old method : 0.03342127799987793 length of segment : 161 time for calcul the mask position with numpy : 0.0019674301147460938 nb_pixel_total : 81152 time to create 1 rle with old method : 0.09863519668579102 length of segment : 370 time for calcul the mask position with numpy : 0.0005140304565429688 nb_pixel_total : 24612 time to create 1 rle with old method : 0.031107664108276367 length of segment : 146 time for calcul the mask position with numpy : 0.00011515617370605469 nb_pixel_total : 4258 time to create 1 rle with old method : 0.0054819583892822266 length of segment : 59 time for calcul the mask position with numpy : 0.0016360282897949219 nb_pixel_total : 71910 time to create 1 rle with old method : 0.08900713920593262 length of segment : 671 time for calcul the mask position with numpy : 0.0018932819366455078 nb_pixel_total : 61159 time to create 1 rle with old method : 0.07349801063537598 length of segment : 636 time for calcul the mask position with numpy : 0.012635946273803711 nb_pixel_total : 494213 time to create 1 rle with new method : 0.028391122817993164 length of segment : 1125 time for calcul the mask position with numpy : 0.005274534225463867 nb_pixel_total : 304690 time to create 1 rle with new method : 0.01107931137084961 length of segment : 829 time for calcul the mask position with numpy : 0.00017189979553222656 nb_pixel_total : 3020 time to create 1 rle with old method : 0.003831148147583008 length of segment : 75 time for calcul the mask position with numpy : 0.0003504753112792969 nb_pixel_total : 19576 time to create 1 rle with old method : 0.023391008377075195 length of segment : 132 time for calcul the mask position with numpy : 0.0004742145538330078 nb_pixel_total : 14275 time to create 1 rle with old method : 0.01706695556640625 length of segment : 222 time for calcul the mask position with numpy : 0.0003352165222167969 nb_pixel_total : 21626 time to create 1 rle with old method : 0.02609395980834961 length of segment : 130 time for calcul the mask position with numpy : 0.0004971027374267578 nb_pixel_total : 8139 time to create 1 rle with old method : 0.010004520416259766 length of segment : 141 time for calcul the mask position with numpy : 0.0010492801666259766 nb_pixel_total : 39436 time to create 1 rle with old method : 0.04697155952453613 length of segment : 177 time for calcul the mask position with numpy : 0.0005671977996826172 nb_pixel_total : 18692 time to create 1 rle with old method : 0.02183842658996582 length of segment : 116 time for calcul the mask position with numpy : 0.0017669200897216797 nb_pixel_total : 88766 time to create 1 rle with old method : 0.10480213165283203 length of segment : 620 time for calcul the mask position with numpy : 0.0002803802490234375 nb_pixel_total : 6490 time to create 1 rle with old method : 0.00819540023803711 length of segment : 104 time for calcul the mask position with numpy : 0.00024580955505371094 nb_pixel_total : 6744 time to create 1 rle with old method : 0.008388042449951172 length of segment : 108 time for calcul the mask position with numpy : 0.0036530494689941406 nb_pixel_total : 196947 time to create 1 rle with new method : 0.0075185298919677734 length of segment : 511 time for calcul the mask position with numpy : 0.0008347034454345703 nb_pixel_total : 36765 time to create 1 rle with old method : 0.043431997299194336 length of segment : 268 time for calcul the mask position with numpy : 0.0032358169555664062 nb_pixel_total : 108051 time to create 1 rle with old method : 0.12253141403198242 length of segment : 552 time for calcul the mask position with numpy : 0.001094818115234375 nb_pixel_total : 16328 time to create 1 rle with old method : 0.019470930099487305 length of segment : 238 time for calcul the mask position with numpy : 0.0015425682067871094 nb_pixel_total : 35442 time to create 1 rle with old method : 0.0420529842376709 length of segment : 295 time for calcul the mask position with numpy : 0.0012254714965820312 nb_pixel_total : 58890 time to create 1 rle with old method : 0.06938648223876953 length of segment : 388 time for calcul the mask position with numpy : 0.0015859603881835938 nb_pixel_total : 105855 time to create 1 rle with old method : 0.12401676177978516 length of segment : 439 time for calcul the mask position with numpy : 0.0014786720275878906 nb_pixel_total : 30241 time to create 1 rle with old method : 0.03553152084350586 length of segment : 367 time for calcul the mask position with numpy : 0.0006575584411621094 nb_pixel_total : 22865 time to create 1 rle with old method : 0.027556419372558594 length of segment : 141 time for calcul the mask position with numpy : 0.0008907318115234375 nb_pixel_total : 31843 time to create 1 rle with old method : 0.039350271224975586 length of segment : 152 time for calcul the mask position with numpy : 0.0013914108276367188 nb_pixel_total : 35491 time to create 1 rle with old method : 0.04469561576843262 length of segment : 232 time for calcul the mask position with numpy : 0.011568784713745117 nb_pixel_total : 478994 time to create 1 rle with new method : 0.01537466049194336 length of segment : 868 time for calcul the mask position with numpy : 0.0005018711090087891 nb_pixel_total : 8674 time to create 1 rle with old method : 0.014546394348144531 length of segment : 110 time for calcul the mask position with numpy : 0.002896547317504883 nb_pixel_total : 75396 time to create 1 rle with old method : 0.09660506248474121 length of segment : 402 time for calcul the mask position with numpy : 0.0005829334259033203 nb_pixel_total : 10045 time to create 1 rle with old method : 0.012181282043457031 length of segment : 233 time for calcul the mask position with numpy : 9.775161743164062e-05 nb_pixel_total : 2938 time to create 1 rle with old method : 0.0038907527923583984 length of segment : 52 time for calcul the mask position with numpy : 0.00026988983154296875 nb_pixel_total : 3499 time to create 1 rle with old method : 0.004419088363647461 length of segment : 127 time for calcul the mask position with numpy : 0.0030062198638916016 nb_pixel_total : 132419 time to create 1 rle with old method : 0.16261720657348633 length of segment : 571 time for calcul the mask position with numpy : 0.0006656646728515625 nb_pixel_total : 17053 time to create 1 rle with old method : 0.020937442779541016 length of segment : 184 time for calcul the mask position with numpy : 0.0018115043640136719 nb_pixel_total : 128942 time to create 1 rle with old method : 0.14909934997558594 length of segment : 536 time for calcul the mask position with numpy : 0.00015163421630859375 nb_pixel_total : 1718 time to create 1 rle with old method : 0.002298116683959961 length of segment : 56 time for calcul the mask position with numpy : 0.00016236305236816406 nb_pixel_total : 1489 time to create 1 rle with old method : 0.0019459724426269531 length of segment : 69 time for calcul the mask position with numpy : 0.000736236572265625 nb_pixel_total : 16231 time to create 1 rle with old method : 0.019323110580444336 length of segment : 193 time for calcul the mask position with numpy : 0.0002357959747314453 nb_pixel_total : 3275 time to create 1 rle with old method : 0.0041980743408203125 length of segment : 119 time for calcul the mask position with numpy : 0.0004818439483642578 nb_pixel_total : 14657 time to create 1 rle with old method : 0.01841449737548828 length of segment : 121 time for calcul the mask position with numpy : 0.0020055770874023438 nb_pixel_total : 45951 time to create 1 rle with old method : 0.05609893798828125 length of segment : 392 time for calcul the mask position with numpy : 0.0005431175231933594 nb_pixel_total : 21682 time to create 1 rle with old method : 0.026972293853759766 length of segment : 234 time for calcul the mask position with numpy : 0.0007867813110351562 nb_pixel_total : 36278 time to create 1 rle with old method : 0.04445672035217285 length of segment : 219 time for calcul the mask position with numpy : 0.00017571449279785156 nb_pixel_total : 1838 time to create 1 rle with old method : 0.002430438995361328 length of segment : 63 time for calcul the mask position with numpy : 0.0005819797515869141 nb_pixel_total : 29817 time to create 1 rle with old method : 0.03739333152770996 length of segment : 191 time for calcul the mask position with numpy : 0.39461469650268555 nb_pixel_total : 4147380 time to create 1 rle with new method : 0.37244319915771484 length of segment : 2687 time for calcul the mask position with numpy : 0.006833314895629883 nb_pixel_total : 480708 time to create 1 rle with new method : 0.028421640396118164 length of segment : 692 time for calcul the mask position with numpy : 0.1393423080444336 nb_pixel_total : 2070995 time to create 1 rle with new method : 0.27938151359558105 length of segment : 3329 time for calcul the mask position with numpy : 0.0005717277526855469 nb_pixel_total : 13926 time to create 1 rle with old method : 0.017370939254760742 length of segment : 134 time for calcul the mask position with numpy : 0.00023293495178222656 nb_pixel_total : 10329 time to create 1 rle with old method : 0.013174057006835938 length of segment : 119 time for calcul the mask position with numpy : 0.0018131732940673828 nb_pixel_total : 100183 time to create 1 rle with old method : 0.11822223663330078 length of segment : 729 time for calcul the mask position with numpy : 0.001958131790161133 nb_pixel_total : 51028 time to create 1 rle with old method : 0.06103777885437012 length of segment : 660 time for calcul the mask position with numpy : 0.0007641315460205078 nb_pixel_total : 36034 time to create 1 rle with old method : 0.04378676414489746 length of segment : 280 time for calcul the mask position with numpy : 0.0005962848663330078 nb_pixel_total : 32230 time to create 1 rle with old method : 0.039466142654418945 length of segment : 276 time for calcul the mask position with numpy : 0.00040793418884277344 nb_pixel_total : 10466 time to create 1 rle with old method : 0.013237476348876953 length of segment : 238 time for calcul the mask position with numpy : 0.00763392448425293 nb_pixel_total : 444444 time to create 1 rle with new method : 0.01603388786315918 length of segment : 1089 time for calcul the mask position with numpy : 0.00024390220642089844 nb_pixel_total : 6041 time to create 1 rle with old method : 0.00965738296508789 length of segment : 102 time for calcul the mask position with numpy : 0.0008854866027832031 nb_pixel_total : 30310 time to create 1 rle with old method : 0.03719305992126465 length of segment : 298 time for calcul the mask position with numpy : 0.00014710426330566406 nb_pixel_total : 2899 time to create 1 rle with old method : 0.0036954879760742188 length of segment : 83 time for calcul the mask position with numpy : 0.003932476043701172 nb_pixel_total : 251706 time to create 1 rle with new method : 0.007544040679931641 length of segment : 529 time for calcul the mask position with numpy : 0.0001201629638671875 nb_pixel_total : 1024 time to create 1 rle with old method : 0.0014803409576416016 length of segment : 48 time spent for convertir_results : 17.314810276031494 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 131 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Number RLEs to save : 51458 save missing photos in datou_result : time spend for datou_step_exec : 65.23124170303345 time spend to save output : 7.7217934131622314 total time spend for step 1 : 72.95303511619568 step2:crop_condition Fri Feb 28 03:19:43 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 131 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 10 About to insert : list_path_to_insert length 10 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 29 About to insert : list_path_to_insert length 29 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1340018829, 1340018826, 1340018786, 1340018755, 1339929112, 1339928413, 1339928409, 1339928405, 1339928400, 1339928322] Looping around the photos to save general results len do output : 62 /-3689358395Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689358392Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689367976Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689455685Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689367994Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368000Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368013Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368035Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368030Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368025Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689358388Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689358396Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689367986Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368012Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368003Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368046Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368034Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689358399Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689455687Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368020Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368018Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368042Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689358385Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689358386Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689358377Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689358384Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689358393Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689358401Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689358389Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689358402Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689358403Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689358406Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689358398Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689358391Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689367983Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689367989Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689455689Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689367998Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368001Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368019Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689455692Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368024Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368022Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368015Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368043Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368029Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368044Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368039Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368048Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368059Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368051Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689358381Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689358390Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689358400Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689367999Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368009Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368027Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368056Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368050Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368036Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689358387Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3689368058Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018829', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018826', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018786', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018755', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339929112', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928413', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928409', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928405', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928400', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928322', None, None, None, None, None, '2621865') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 196 time used for this insertion : 0.033524513244628906 save_final save missing photos in datou_result : time spend for datou_step_exec : 32.38470721244812 time spend to save output : 0.036359310150146484 total time spend for step 2 : 32.42106652259827 step3:thcl Fri Feb 28 03:20:15 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.05348539352416992 time to convert the images to numpy array : 3.0994415283203125e-06 time to import caffe and check if the image exist : 0.05704331398010254 time to convert the images to numpy array : 0.036968231201171875 time to import caffe and check if the image exist : 0.05899524688720703 time to convert the images to numpy array : 0.03928947448730469 time to import caffe and check if the image exist : 0.054213762283325195 time to convert the images to numpy array : 0.06470608711242676 time to import caffe and check if the image exist : 0.057403564453125 time to convert the images to numpy array : 0.16252517700195312 time to import caffe and check if the image exist : 0.05764889717102051 time to convert the images to numpy array : 0.16645050048828125 time to import caffe and check if the image exist : 0.06019091606140137 time to convert the images to numpy array : 0.20371699333190918 time to import caffe and check if the image exist : 0.06109499931335449 time to convert the images to numpy array : 0.20947670936584473 time to import caffe and check if the image exist : 0.05834794044494629 time to convert the images to numpy array : 0.21651721000671387 time to import caffe and check if the image exist : 0.059885501861572266 time to convert the images to numpy array : 0.7114019393920898 total time to convert the images to numpy array : 0.9674196243286133 list photo_ids error: [] list photo_ids correct : [-3689368059, -3689368051, -3689358381, -3689358390, -3689358400, -3689367999, -3689368009, -3689368027, -3689368056, -3689368050, -3689368036, -3689358387, -3689368058, -3689358395, -3689358392, -3689367976, -3689455685, -3689367994, -3689368000, -3689368013, -3689368035, -3689368030, -3689368025, -3689358388, -3689358396, -3689367986, -3689368012, -3689358389, -3689358402, -3689358403, -3689358406, -3689358398, -3689358391, -3689367983, -3689367989, -3689455689, -3689367998, -3689368001, -3689368019, -3689455692, -3689368024, -3689368003, -3689368046, -3689368034, -3689358399, -3689455687, -3689368020, -3689368018, -3689368022, -3689368015, -3689368043, -3689368029, -3689368044, -3689368039, -3689368048, -3689368042, -3689358385, -3689358386, -3689358377, -3689358384, -3689358393, -3689358401] number of photos to traite : 62 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5304 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5085 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.7078907489776611 time used to do the prediction : 0.1645045280456543 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5083 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5083 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.7109956741333008 time used to do the prediction : 0.1283416748046875 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 2.3603439331054688e-05 save missing photos in datou_result : time spend for datou_step_exec : 9.235771894454956 time spend to save output : 0.0004990100860595703 total time spend for step 3 : 9.236270904541016 step4:merge_mask_thcl_custom Fri Feb 28 03:20:24 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 131 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1340018829, 1340018826, 1340018786, 1340018755, 1339929112, 1339928413, 1339928409, 1339928405, 1339928400, 1339928322] Looping around the photos to save general results len do output : 10 /1340018829Didn't retrieve data .Didn't retrieve data . /1340018826Didn't retrieve data .Didn't retrieve data . /1340018786Didn't retrieve data .Didn't retrieve data . /1340018755Didn't retrieve data .Didn't retrieve data . /1339929112Didn't retrieve data .Didn't retrieve data . /1339928413Didn't retrieve data .Didn't retrieve data . /1339928409Didn't retrieve data .Didn't retrieve data . /1339928405Didn't retrieve data .Didn't retrieve data . /1339928400Didn't retrieve data .Didn't retrieve data . /1339928322Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018829', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018826', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018786', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018755', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339929112', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928413', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928409', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928405', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928400', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928322', None, None, None, None, None, '2621865') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.014493942260742188 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.06460833549499512 time spend to save output : 0.015016555786132812 total time spend for step 4 : 0.07962489128112793 step5:rle_unique_nms_with_priority Fri Feb 28 03:20:25 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 12 nb_hashtags : 5 time to prepare the origin masks : 9.454254627227783 time for calcul the mask position with numpy : 0.8026363849639893 nb_pixel_total : 6583780 time to create 1 rle with new method : 1.2666139602661133 time for calcul the mask position with numpy : 0.03422856330871582 nb_pixel_total : 15256 time to create 1 rle with old method : 0.01869988441467285 time for calcul the mask position with numpy : 0.03496265411376953 nb_pixel_total : 15055 time to create 1 rle with old method : 0.01733231544494629 time for calcul the mask position with numpy : 0.0315396785736084 nb_pixel_total : 52754 time to create 1 rle with old method : 0.06105637550354004 time for calcul the mask position with numpy : 0.023819446563720703 nb_pixel_total : 78581 time to create 1 rle with old method : 0.10756707191467285 time for calcul the mask position with numpy : 0.024107933044433594 nb_pixel_total : 49360 time to create 1 rle with old method : 0.0570223331451416 time for calcul the mask position with numpy : 0.022736549377441406 nb_pixel_total : 12954 time to create 1 rle with old method : 0.015267372131347656 time for calcul the mask position with numpy : 0.02373051643371582 nb_pixel_total : 6260 time to create 1 rle with old method : 0.0074005126953125 time for calcul the mask position with numpy : 0.025762557983398438 nb_pixel_total : 12812 time to create 1 rle with old method : 0.01514291763305664 time for calcul the mask position with numpy : 0.02832770347595215 nb_pixel_total : 6499 time to create 1 rle with old method : 0.007496833801269531 time for calcul the mask position with numpy : 0.0266416072845459 nb_pixel_total : 8003 time to create 1 rle with old method : 0.009286642074584961 time for calcul the mask position with numpy : 0.025166988372802734 nb_pixel_total : 972 time to create 1 rle with old method : 0.0011911392211914062 time for calcul the mask position with numpy : 0.027288436889648438 nb_pixel_total : 207954 time to create 1 rle with new method : 1.1348373889923096 create new chi : 3.9090254306793213 time to delete rle : 0.0152130126953125 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 7034 TO DO : save crop sub photo not yet done ! save time : 0.5023531913757324 nb_obj : 18 nb_hashtags : 7 time to prepare the origin masks : 14.053975343704224 time for calcul the mask position with numpy : 0.34709668159484863 nb_pixel_total : 5602220 time to create 1 rle with new method : 0.5335776805877686 time for calcul the mask position with numpy : 0.0354151725769043 nb_pixel_total : 54148 time to create 1 rle with old method : 0.06133294105529785 time for calcul the mask position with numpy : 0.03355240821838379 nb_pixel_total : 128165 time to create 1 rle with old method : 0.14537858963012695 time for calcul the mask position with numpy : 0.02655935287475586 nb_pixel_total : 355917 time to create 1 rle with new method : 0.6322674751281738 time for calcul the mask position with numpy : 0.02263641357421875 nb_pixel_total : 46613 time to create 1 rle with old method : 0.05307602882385254 time for calcul the mask position with numpy : 0.022688865661621094 nb_pixel_total : 9953 time to create 1 rle with old method : 0.011281251907348633 time for calcul the mask position with numpy : 0.026237964630126953 nb_pixel_total : 22990 time to create 1 rle with old method : 0.03150629997253418 time for calcul the mask position with numpy : 0.036992549896240234 nb_pixel_total : 150184 time to create 1 rle with new method : 0.5740518569946289 time for calcul the mask position with numpy : 0.0393831729888916 nb_pixel_total : 11677 time to create 1 rle with old method : 0.01968836784362793 time for calcul the mask position with numpy : 0.03520059585571289 nb_pixel_total : 3100 time to create 1 rle with old method : 0.0035839080810546875 time for calcul the mask position with numpy : 0.03447747230529785 nb_pixel_total : 6969 time to create 1 rle with old method : 0.008170127868652344 time for calcul the mask position with numpy : 0.03628420829772949 nb_pixel_total : 281591 time to create 1 rle with new method : 0.6616199016571045 time for calcul the mask position with numpy : 0.03352999687194824 nb_pixel_total : 31168 time to create 1 rle with old method : 0.03632473945617676 time for calcul the mask position with numpy : 0.036748647689819336 nb_pixel_total : 1421 time to create 1 rle with old method : 0.001692056655883789 time for calcul the mask position with numpy : 0.028997182846069336 nb_pixel_total : 96700 time to create 1 rle with old method : 0.11462783813476562 time for calcul the mask position with numpy : 0.021227121353149414 nb_pixel_total : 4178 time to create 1 rle with old method : 0.0048143863677978516 time for calcul the mask position with numpy : 0.022611379623413086 nb_pixel_total : 161712 time to create 1 rle with new method : 0.540642261505127 time for calcul the mask position with numpy : 0.022012948989868164 nb_pixel_total : 22207 time to create 1 rle with old method : 0.024901151657104492 time for calcul the mask position with numpy : 0.021568775177001953 nb_pixel_total : 59327 time to create 1 rle with old method : 0.06690406799316406 create new chi : 4.541489124298096 time to delete rle : 0.002153635025024414 batch 1 Loaded 19 chid ids of type : 3726 Number RLEs to save : 15508 TO DO : save crop sub photo not yet done ! save time : 1.5895037651062012 nb_obj : 1 nb_hashtags : 1 time to prepare the origin masks : 1.3742289543151855 time for calcul the mask position with numpy : 0.41411280632019043 nb_pixel_total : 3204485 time to create 1 rle with new method : 1.6947481632232666 time for calcul the mask position with numpy : 0.2595560550689697 nb_pixel_total : 3845755 time to create 1 rle with new method : 1.150052547454834 create new chi : 3.587348222732544 time to delete rle : 0.0013234615325927734 batch 1 Loaded 2 chid ids of type : 3726 Number RLEs to save : 8966 TO DO : save crop sub photo not yet done ! save time : 0.538165807723999 nb_obj : 14 nb_hashtags : 4 time to prepare the origin masks : 8.920088529586792 time for calcul the mask position with numpy : 0.22610259056091309 nb_pixel_total : 5725311 time to create 1 rle with new method : 0.449352502822876 time for calcul the mask position with numpy : 0.02138996124267578 nb_pixel_total : 35439 time to create 1 rle with old method : 0.04286456108093262 time for calcul the mask position with numpy : 0.023075342178344727 nb_pixel_total : 4853 time to create 1 rle with old method : 0.005549430847167969 time for calcul the mask position with numpy : 0.022695064544677734 nb_pixel_total : 39179 time to create 1 rle with old method : 0.04620003700256348 time for calcul the mask position with numpy : 0.023310184478759766 nb_pixel_total : 3641 time to create 1 rle with old method : 0.004258632659912109 time for calcul the mask position with numpy : 0.023449420928955078 nb_pixel_total : 49819 time to create 1 rle with old method : 0.06328964233398438 time for calcul the mask position with numpy : 0.023942232131958008 nb_pixel_total : 2881 time to create 1 rle with old method : 0.003483295440673828 time for calcul the mask position with numpy : 0.02646327018737793 nb_pixel_total : 237249 time to create 1 rle with new method : 0.8373396396636963 time for calcul the mask position with numpy : 0.021730422973632812 nb_pixel_total : 31253 time to create 1 rle with old method : 0.035140275955200195 time for calcul the mask position with numpy : 0.023500442504882812 nb_pixel_total : 2805 time to create 1 rle with old method : 0.004204750061035156 time for calcul the mask position with numpy : 0.02974724769592285 nb_pixel_total : 607905 time to create 1 rle with new method : 0.5431621074676514 time for calcul the mask position with numpy : 0.025037527084350586 nb_pixel_total : 10417 time to create 1 rle with old method : 0.013589859008789062 time for calcul the mask position with numpy : 0.02372145652770996 nb_pixel_total : 99594 time to create 1 rle with old method : 0.12025904655456543 time for calcul the mask position with numpy : 0.037868499755859375 nb_pixel_total : 190777 time to create 1 rle with new method : 1.1076343059539795 time for calcul the mask position with numpy : 0.043694257736206055 nb_pixel_total : 9117 time to create 1 rle with old method : 0.010972976684570312 create new chi : 4.002742528915405 time to delete rle : 0.0019259452819824219 batch 1 Loaded 20 chid ids of type : 3726 Number RLEs to save : 13701 TO DO : save crop sub photo not yet done ! save time : 1.0879013538360596 nb_obj : 8 nb_hashtags : 3 time to prepare the origin masks : 5.955408096313477 time for calcul the mask position with numpy : 0.8412787914276123 nb_pixel_total : 4207800 time to create 1 rle with new method : 1.2145118713378906 time for calcul the mask position with numpy : 0.054343223571777344 nb_pixel_total : 1264 time to create 1 rle with old method : 0.0030748844146728516 time for calcul the mask position with numpy : 0.063934326171875 nb_pixel_total : 300830 time to create 1 rle with new method : 0.8220930099487305 time for calcul the mask position with numpy : 0.03838706016540527 nb_pixel_total : 97424 time to create 1 rle with old method : 0.10890769958496094 time for calcul the mask position with numpy : 0.03799271583557129 nb_pixel_total : 67531 time to create 1 rle with old method : 0.07550978660583496 time for calcul the mask position with numpy : 0.04012489318847656 nb_pixel_total : 220830 time to create 1 rle with new method : 0.46040940284729004 time for calcul the mask position with numpy : 0.03577399253845215 nb_pixel_total : 1454 time to create 1 rle with old method : 0.0017304420471191406 time for calcul the mask position with numpy : 0.034177541732788086 nb_pixel_total : 12841 time to create 1 rle with old method : 0.020338058471679688 time for calcul the mask position with numpy : 0.11141204833984375 nb_pixel_total : 2140266 time to create 1 rle with new method : 0.6311249732971191 create new chi : 4.724493503570557 time to delete rle : 0.0022749900817871094 batch 1 Loaded 11 chid ids of type : 3726 Number RLEs to save : 15448 TO DO : save crop sub photo not yet done ! save time : 4.11479377746582 nb_obj : 19 nb_hashtags : 7 time to prepare the origin masks : 21.499881505966187 time for calcul the mask position with numpy : 2.849307060241699 nb_pixel_total : 5695621 time to create 1 rle with new method : 0.9226293563842773 time for calcul the mask position with numpy : 0.2648177146911621 nb_pixel_total : 8146 time to create 1 rle with old method : 0.009979248046875 time for calcul the mask position with numpy : 0.5737340450286865 nb_pixel_total : 26496 time to create 1 rle with old method : 0.10845518112182617 time for calcul the mask position with numpy : 0.5938565731048584 nb_pixel_total : 5122 time to create 1 rle with old method : 0.02024078369140625 time for calcul the mask position with numpy : 0.7210037708282471 nb_pixel_total : 19534 time to create 1 rle with old method : 0.029332876205444336 time for calcul the mask position with numpy : 0.393951416015625 nb_pixel_total : 14243 time to create 1 rle with old method : 0.017971277236938477 time for calcul the mask position with numpy : 0.5103979110717773 nb_pixel_total : 81034 time to create 1 rle with old method : 0.09660458564758301 time for calcul the mask position with numpy : 0.3625659942626953 nb_pixel_total : 24514 time to create 1 rle with old method : 0.031625986099243164 time for calcul the mask position with numpy : 0.03643441200256348 nb_pixel_total : 3005 time to create 1 rle with old method : 0.003575563430786133 time for calcul the mask position with numpy : 0.08720707893371582 nb_pixel_total : 243988 time to create 1 rle with new method : 0.6978409290313721 time for calcul the mask position with numpy : 0.07185697555541992 nb_pixel_total : 493868 time to create 1 rle with new method : 0.7515077590942383 time for calcul the mask position with numpy : 0.03301405906677246 nb_pixel_total : 60838 time to create 1 rle with old method : 0.07518863677978516 time for calcul the mask position with numpy : 0.08989810943603516 nb_pixel_total : 3097 time to create 1 rle with old method : 0.023627042770385742 time for calcul the mask position with numpy : 0.024019718170166016 nb_pixel_total : 4234 time to create 1 rle with old method : 0.0056171417236328125 time for calcul the mask position with numpy : 0.02499699592590332 nb_pixel_total : 82105 time to create 1 rle with old method : 0.11784124374389648 time for calcul the mask position with numpy : 0.02865743637084961 nb_pixel_total : 31779 time to create 1 rle with old method : 0.048877716064453125 time for calcul the mask position with numpy : 0.02399277687072754 nb_pixel_total : 7000 time to create 1 rle with old method : 0.009389400482177734 time for calcul the mask position with numpy : 0.08345222473144531 nb_pixel_total : 127794 time to create 1 rle with old method : 0.18059158325195312 time for calcul the mask position with numpy : 0.02314615249633789 nb_pixel_total : 3823 time to create 1 rle with old method : 0.0051975250244140625 time for calcul the mask position with numpy : 0.02357625961303711 nb_pixel_total : 113999 time to create 1 rle with old method : 0.29177355766296387 create new chi : 10.405519962310791 time to delete rle : 0.10724902153015137 batch 1 Loaded 21 chid ids of type : 3726 Number RLEs to save : 15357 TO DO : save crop sub photo not yet done ! save time : 1.4738781452178955 nb_obj : 12 nb_hashtags : 6 time to prepare the origin masks : 9.757101774215698 time for calcul the mask position with numpy : 0.4111058712005615 nb_pixel_total : 6333112 time to create 1 rle with new method : 0.778486967086792 time for calcul the mask position with numpy : 0.023026704788208008 nb_pixel_total : 89225 time to create 1 rle with old method : 0.10439586639404297 time for calcul the mask position with numpy : 0.026274681091308594 nb_pixel_total : 196786 time to create 1 rle with new method : 0.5136723518371582 time for calcul the mask position with numpy : 0.022218942642211914 nb_pixel_total : 35347 time to create 1 rle with old method : 0.03895306587219238 time for calcul the mask position with numpy : 0.02422332763671875 nb_pixel_total : 36697 time to create 1 rle with old method : 0.04846477508544922 time for calcul the mask position with numpy : 0.03319072723388672 nb_pixel_total : 107546 time to create 1 rle with old method : 0.13570737838745117 time for calcul the mask position with numpy : 0.03995227813720703 nb_pixel_total : 16248 time to create 1 rle with old method : 0.02011847496032715 time for calcul the mask position with numpy : 0.035157203674316406 nb_pixel_total : 6713 time to create 1 rle with old method : 0.007892847061157227 time for calcul the mask position with numpy : 0.022284269332885742 nb_pixel_total : 58912 time to create 1 rle with old method : 0.0701751708984375 time for calcul the mask position with numpy : 0.025818586349487305 nb_pixel_total : 39228 time to create 1 rle with old method : 0.0523073673248291 time for calcul the mask position with numpy : 0.022350072860717773 nb_pixel_total : 6426 time to create 1 rle with old method : 0.007776498794555664 time for calcul the mask position with numpy : 0.02785015106201172 nb_pixel_total : 18437 time to create 1 rle with old method : 0.022438764572143555 time for calcul the mask position with numpy : 0.029150962829589844 nb_pixel_total : 105563 time to create 1 rle with old method : 0.13279128074645996 create new chi : 2.991957664489746 time to delete rle : 0.0011935234069824219 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 9758 TO DO : save crop sub photo not yet done ! save time : 1.2461752891540527 nb_obj : 21 nb_hashtags : 7 time to prepare the origin masks : 12.35172152519226 time for calcul the mask position with numpy : 0.3884010314941406 nb_pixel_total : 6157385 time to create 1 rle with new method : 0.9487853050231934 time for calcul the mask position with numpy : 0.03594851493835449 nb_pixel_total : 3127 time to create 1 rle with old method : 0.004018068313598633 time for calcul the mask position with numpy : 0.03694629669189453 nb_pixel_total : 16190 time to create 1 rle with old method : 0.018459320068359375 time for calcul the mask position with numpy : 0.03809499740600586 nb_pixel_total : 132204 time to create 1 rle with old method : 0.15086007118225098 time for calcul the mask position with numpy : 0.03707098960876465 nb_pixel_total : 75283 time to create 1 rle with old method : 0.08582711219787598 time for calcul the mask position with numpy : 0.04022049903869629 nb_pixel_total : 4988 time to create 1 rle with old method : 0.0057637691497802734 time for calcul the mask position with numpy : 0.03928017616271973 nb_pixel_total : 1832 time to create 1 rle with old method : 0.0034377574920654297 time for calcul the mask position with numpy : 0.03804588317871094 nb_pixel_total : 29737 time to create 1 rle with old method : 0.0364682674407959 time for calcul the mask position with numpy : 0.04195761680603027 nb_pixel_total : 3263 time to create 1 rle with old method : 0.004345417022705078 time for calcul the mask position with numpy : 0.039383888244628906 nb_pixel_total : 1484 time to create 1 rle with old method : 0.0017919540405273438 time for calcul the mask position with numpy : 0.035018205642700195 nb_pixel_total : 1708 time to create 1 rle with old method : 0.0022084712982177734 time for calcul the mask position with numpy : 0.03804445266723633 nb_pixel_total : 16997 time to create 1 rle with old method : 0.0188291072845459 time for calcul the mask position with numpy : 0.03430581092834473 nb_pixel_total : 57 time to create 1 rle with old method : 0.00034236907958984375 time for calcul the mask position with numpy : 0.021233797073364258 nb_pixel_total : 4744 time to create 1 rle with old method : 0.006072998046875 time for calcul the mask position with numpy : 0.021576881408691406 nb_pixel_total : 8642 time to create 1 rle with old method : 0.009901046752929688 time for calcul the mask position with numpy : 0.021335363388061523 nb_pixel_total : 3496 time to create 1 rle with old method : 0.004064321517944336 time for calcul the mask position with numpy : 0.02088451385498047 nb_pixel_total : 327 time to create 1 rle with old method : 0.00044536590576171875 time for calcul the mask position with numpy : 0.020833730697631836 nb_pixel_total : 10038 time to create 1 rle with old method : 0.011892080307006836 time for calcul the mask position with numpy : 0.026818037033081055 nb_pixel_total : 478570 time to create 1 rle with new method : 0.39813995361328125 time for calcul the mask position with numpy : 0.021898269653320312 nb_pixel_total : 22741 time to create 1 rle with old method : 0.028713226318359375 time for calcul the mask position with numpy : 0.032080650329589844 nb_pixel_total : 31581 time to create 1 rle with old method : 0.05657458305358887 time for calcul the mask position with numpy : 0.021651029586791992 nb_pixel_total : 45846 time to create 1 rle with old method : 0.05828976631164551 create new chi : 2.9731836318969727 time to delete rle : 0.0011239051818847656 batch 1 Loaded 23 chid ids of type : 3726 Number RLEs to save : 11047 TO DO : save crop sub photo not yet done ! save time : 3.815957546234131 nb_obj : 3 nb_hashtags : 2 time to prepare the origin masks : 3.8283121585845947 time for calcul the mask position with numpy : 0.05780458450317383 nb_pixel_total : 2297755 time to create 1 rle with new method : 1.0053961277008057 time for calcul the mask position with numpy : 0.027112960815429688 nb_pixel_total : 479481 time to create 1 rle with new method : 0.7549395561218262 time for calcul the mask position with numpy : 0.03035449981689453 nb_pixel_total : 129289 time to create 1 rle with old method : 0.1593921184539795 time for calcul the mask position with numpy : 0.2267756462097168 nb_pixel_total : 4143715 time to create 1 rle with new method : 0.482957124710083 create new chi : 2.8227877616882324 time to delete rle : 0.0012209415435791016 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 11299 TO DO : save crop sub photo not yet done ! save time : 1.0186994075775146 nb_obj : 14 nb_hashtags : 4 time to prepare the origin masks : 8.30082631111145 time for calcul the mask position with numpy : 0.3603050708770752 nb_pixel_total : 4392164 time to create 1 rle with new method : 1.0617578029632568 time for calcul the mask position with numpy : 0.021152496337890625 nb_pixel_total : 781 time to create 1 rle with old method : 0.001043081283569336 time for calcul the mask position with numpy : 0.021221637725830078 nb_pixel_total : 552 time to create 1 rle with old method : 0.0007588863372802734 time for calcul the mask position with numpy : 0.020874738693237305 nb_pixel_total : 1011 time to create 1 rle with old method : 0.001211404800415039 time for calcul the mask position with numpy : 0.030742883682250977 nb_pixel_total : 67562 time to create 1 rle with old method : 0.08466482162475586 time for calcul the mask position with numpy : 0.022776126861572266 nb_pixel_total : 2886 time to create 1 rle with old method : 0.004717111587524414 time for calcul the mask position with numpy : 0.023956298828125 nb_pixel_total : 243876 time to create 1 rle with new method : 1.060143232345581 time for calcul the mask position with numpy : 0.02245044708251953 nb_pixel_total : 32180 time to create 1 rle with old method : 0.03749990463256836 time for calcul the mask position with numpy : 0.021614551544189453 nb_pixel_total : 13796 time to create 1 rle with old method : 0.015807390213012695 time for calcul the mask position with numpy : 0.022881269454956055 nb_pixel_total : 35898 time to create 1 rle with old method : 0.0461118221282959 time for calcul the mask position with numpy : 0.025195837020874023 nb_pixel_total : 10446 time to create 1 rle with old method : 0.011678218841552734 time for calcul the mask position with numpy : 0.02374577522277832 nb_pixel_total : 51064 time to create 1 rle with old method : 0.06628131866455078 time for calcul the mask position with numpy : 0.023504257202148438 nb_pixel_total : 100149 time to create 1 rle with old method : 0.11489677429199219 time for calcul the mask position with numpy : 0.021416425704956055 nb_pixel_total : 27895 time to create 1 rle with old method : 0.033902645111083984 time for calcul the mask position with numpy : 0.0411069393157959 nb_pixel_total : 2069980 time to create 1 rle with new method : 0.4252011775970459 create new chi : 3.7523040771484375 time to delete rle : 0.0015759468078613281 batch 1 Loaded 15 chid ids of type : 3726 Number RLEs to save : 16541 TO DO : save crop sub photo not yet done ! save time : 1.6687326431274414 map_output_result : {1340018829: (0.0, 'Should be the crop_list due to order', 0), 1340018826: (0.0, 'Should be the crop_list due to order', 0), 1340018786: (0.0, 'Should be the crop_list due to order', 0), 1340018755: (0.0, 'Should be the crop_list due to order', 0), 1339929112: (0.0, 'Should be the crop_list due to order', 0), 1339928413: (0.0, 'Should be the crop_list due to order', 0), 1339928409: (0.0, 'Should be the crop_list due to order', 0), 1339928405: (0.0, 'Should be the crop_list due to order', 0), 1339928400: (0.0, 'Should be the crop_list due to order', 0), 1339928322: (0.0, 'Should be the crop_list due to order', 0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1340018829, 1340018826, 1340018786, 1340018755, 1339929112, 1339928413, 1339928409, 1339928405, 1339928400, 1339928322] Looping around the photos to save general results len do output : 10 /1340018829.Didn't retrieve data . /1340018826.Didn't retrieve data . /1340018786.Didn't retrieve data . /1340018755.Didn't retrieve data . /1339929112.Didn't retrieve data . /1339928413.Didn't retrieve data . /1339928409.Didn't retrieve data . /1339928405.Didn't retrieve data . /1339928400.Didn't retrieve data . /1339928322.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018829', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018826', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018786', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018755', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339929112', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928413', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928409', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928405', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928400', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928322', None, None, None, None, None, '2621865') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013895273208618164 save_final save missing photos in datou_result : time spend for datou_step_exec : 159.148775100708 time spend to save output : 0.022097110748291016 total time spend for step 5 : 159.1708722114563 step6:crop_condition Fri Feb 28 03:23:04 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 144 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! About to upload 8 photos upload in portfolio : 4869462 init cache_photo without model_param we have 8 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740709387_2701424 we have uploaded 8 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.755444049835205 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740709394_2701424 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.8150186538696289 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740709397_2701424 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.376420259475708 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740709400_2701424 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.5931124687194824 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740709404_2701424 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.835472822189331 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740709409_2701424 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.591761589050293 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740709414_2701424 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.912926435470581 we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1340018829, 1340018826, 1340018786, 1340018755, 1339929112, 1339928413, 1339928409, 1339928405, 1339928400, 1339928322] Looping around the photos to save general results len do output : 28 /1340049434Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340049435Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340049436Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340049437Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340049438Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340049439Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340049441Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340049442Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340049452Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340049453Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340049466Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340049468Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340049471Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340049473Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340049477Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340049479Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340049481Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340049492Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340049493Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340049494Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340049495Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340049496Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340049497Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340049506Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340049507Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340049508Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340049511Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340049512Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018829', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018826', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018786', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018755', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339929112', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928413', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928409', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928405', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928400', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928322', None, None, None, None, None, '2621865') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 94 time used for this insertion : 0.03581953048706055 save_final save missing photos in datou_result : time spend for datou_step_exec : 34.748544216156006 time spend to save output : 0.04685544967651367 total time spend for step 6 : 34.79539966583252 step7:ventilate_hashtags_in_portfolio Fri Feb 28 03:23:38 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20905796 get user id for portfolio 20905796 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20905796 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Carton_brun','metal','papier','plastique','mal_croppe','Teint_Dans_La_Masse','flou','autre_refus','environnement','kraft','cartonnette','Carton_gris')) AND mptpi.`min_score`=0.5 To do To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20905796 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Carton_brun','metal','papier','plastique','mal_croppe','Teint_Dans_La_Masse','flou','autre_refus','environnement','kraft','cartonnette','Carton_gris')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20906043,20906044,20906045,20906046,20906047,20906048,20906049,20906050,20906051,20906052,20906053,20906054?tags=Carton_gris,Teint_Dans_La_Masse,papier,Carton_brun,cartonnette,plastique,kraft,metal,environnement,flou,mal_croppe,autre_refus Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1340018829, 1340018826, 1340018786, 1340018755, 1339929112, 1339928413, 1339928409, 1339928405, 1339928400, 1339928322] Looping around the photos to save general results len do output : 1 /20905796. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018829', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018826', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018786', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018755', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339929112', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928413', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928409', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928405', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928400', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928322', None, None, None, None, None, '2621865') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.014499664306640625 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.1054604053497314 time spend to save output : 0.014786005020141602 total time spend for step 7 : 1.120246410369873 step8:final Fri Feb 28 03:23:40 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1340018829: ('0.0666171726367893',), 1340018826: ('0.0666171726367893',), 1340018786: ('0.0666171726367893',), 1340018755: ('0.0666171726367893',), 1339929112: ('0.0666171726367893',), 1339928413: ('0.0666171726367893',), 1339928409: ('0.0666171726367893',), 1339928405: ('0.0666171726367893',), 1339928400: ('0.0666171726367893',), 1339928322: ('0.0666171726367893',)} new output for save of step final : {1340018829: ('0.0666171726367893',), 1340018826: ('0.0666171726367893',), 1340018786: ('0.0666171726367893',), 1340018755: ('0.0666171726367893',), 1339929112: ('0.0666171726367893',), 1339928413: ('0.0666171726367893',), 1339928409: ('0.0666171726367893',), 1339928405: ('0.0666171726367893',), 1339928400: ('0.0666171726367893',), 1339928322: ('0.0666171726367893',)} [1340018829, 1340018826, 1340018786, 1340018755, 1339929112, 1339928413, 1339928409, 1339928405, 1339928400, 1339928322] Looping around the photos to save general results len do output : 10 /1340018829.Didn't retrieve data . /1340018826.Didn't retrieve data . /1340018786.Didn't retrieve data . /1340018755.Didn't retrieve data . /1339929112.Didn't retrieve data . /1339928413.Didn't retrieve data . /1339928409.Didn't retrieve data . /1339928405.Didn't retrieve data . /1339928400.Didn't retrieve data . /1339928322.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018829', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018826', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018786', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018755', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339929112', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928413', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928409', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928405', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928400', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928322', None, None, None, None, None, '2621865') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.018416643142700195 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.15589165687561035 time spend to save output : 0.019144296646118164 total time spend for step 8 : 0.17503595352172852 step9:velours_tree Fri Feb 28 03:23:40 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.27277302742004395 time spend to save output : 5.7220458984375e-05 total time spend for step 9 : 0.2728302478790283 step10:send_mail_cod Fri Feb 28 03:23:40 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20905796_28-02-2025_03_23_40.pdf 20906047 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette209060471740709420 20906048 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette209060481740709421 20906049 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette209060491740709421 20906050 imagette209060501740709422 20906052 imagette209060521740709422 20906053 imagette209060531740709422 20906054 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette209060541740709422 20906043 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette209060431740709423 20906044 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette209060441740709423 20906046 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette209060461740709425 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20905796 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20906043,20906044,20906045,20906046,20906047,20906048,20906049,20906050,20906051,20906052,20906053,20906054?tags=Carton_gris,Teint_Dans_La_Masse,papier,Carton_brun,cartonnette,plastique,kraft,metal,environnement,flou,mal_croppe,autre_refus your option no_mail is active, we will not send the real mail to your client args[1340018829] : ('0.0666171726367893',) no score found for photo 1340018829 We are sending mail with results at report@fotonower.com args[1340018826] : ('0.0666171726367893',) no score found for photo 1340018826 We are sending mail with results at report@fotonower.com args[1340018786] : ('0.0666171726367893',) no score found for photo 1340018786 We are sending mail with results at report@fotonower.com args[1340018755] : ('0.0666171726367893',) no score found for photo 1340018755 We are sending mail with results at report@fotonower.com args[1339929112] : ('0.0666171726367893',) no score found for photo 1339929112 We are sending mail with results at report@fotonower.com args[1339928413] : ('0.0666171726367893',) no score found for photo 1339928413 We are sending mail with results at report@fotonower.com args[1339928409] : ('0.0666171726367893',) no score found for photo 1339928409 We are sending mail with results at report@fotonower.com args[1339928405] : ('0.0666171726367893',) no score found for photo 1339928405 We are sending mail with results at report@fotonower.com args[1339928400] : ('0.0666171726367893',) no score found for photo 1339928400 We are sending mail with results at report@fotonower.com args[1339928322] : ('0.0666171726367893',) no score found for photo 1339928322 We are sending mail with results at report@fotonower.com refus_total : 0.0666171726367893 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20905796 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1340018826,1340026473,1340026456,1340026452,1340026110,1340026109,1340018842,1339928322,1340018829,1340026513,1340026522,1339929112,1339928413,1339928409,1339928405,1340018836,1340026538,1340018786,1340018755,1339928400) Found this number of photos: 20 begin to download photo : 1340018826 begin to download photo : 1340026109 begin to download photo : 1340026522 begin to download photo : 1340018836 download finish for photo 1340018826 begin to download photo : 1340026473 download finish for photo 1340026109 begin to download photo : 1340018842 download finish for photo 1340026522 begin to download photo : 1339929112 download finish for photo 1340018836 begin to download photo : 1340026538 download finish for photo 1340026473 begin to download photo : 1340026456 download finish for photo 1340018842 begin to download photo : 1339928322 download finish for photo 1339929112 begin to download photo : 1339928413 download finish for photo 1340026538 begin to download photo : 1340018786 download finish for photo 1340026456 begin to download photo : 1340026452 download finish for photo 1339928322 begin to download photo : 1340018829 download finish for photo 1339928413 begin to download photo : 1339928409 download finish for photo 1340018786 begin to download photo : 1340018755 download finish for photo 1340026452 begin to download photo : 1340026110 download finish for photo 1340018829 begin to download photo : 1340026513 download finish for photo 1339928409 begin to download photo : 1339928405 download finish for photo 1340018755 begin to download photo : 1339928400 download finish for photo 1340026110 download finish for photo 1339928400 download finish for photo 1339928405 download finish for photo 1340026513 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20905796_28-02-2025_03_23_40.pdf results_Auto_P20905796_28-02-2025_03_23_40.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20905796_28-02-2025_03_23_40.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20905796','results_Auto_P20905796_28-02-2025_03_23_40.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20905796_28-02-2025_03_23_40.pdf','pdf','','1.0','0.0666171726367893') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1340018829, 1340018826, 1340018786, 1340018755, 1339929112, 1339928413, 1339928409, 1339928405, 1339928400, 1339928322] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018829', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018826', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018786', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018755', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339929112', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928413', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928409', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928405', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928400', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928322', None, None, None, None, None, '2621865') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.016207218170166016 save_final save missing photos in datou_result : time spend for datou_step_exec : 11.440619945526123 time spend to save output : 0.01646876335144043 total time spend for step 10 : 11.457088708877563 step11:split_time_score Fri Feb 28 03:23:52 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('15', 12), ('13', 8)) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 27022025 20905796 Nombre de photos uploadées : 20 / 23040 (0%) 27022025 20905796 Nombre de photos taguées (types de déchets): 0 / 20 (0%) 27022025 20905796 Nombre de photos taguées (volume) : 0 / 20 (0%) elapsed_time : load_data_split_time_score 2.1457672119140625e-06 elapsed_time : order_list_meta_photo_and_scores 7.3909759521484375e-06 ???????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0013659000396728516 elapsed_time : insert_dashboard_record_day_entry 0.02505636215209961 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901267 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901268 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901269 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901270 order by id desc limit 1 Qualite : 0.07406922604400437 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901271_28-02-2025_02_07_10.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901271 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20901271 AND mptpi.`type`=4207 To do Qualite : 0.08224904095108558 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901272_28-02-2025_01_50_25.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901272 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20901272 AND mptpi.`type`=4207 To do Qualite : 0.13080982673511443 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901273_28-02-2025_03_19_56.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901273 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11560 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11567 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11567 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11563 crop_condition is not consistent : 4 used against 2 in the step definition ! Step 11563 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11564 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11564 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11573 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11573 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11566 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11566 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 1 of step 11560 have datatype=2 whereas input 1 of step 11564 have datatype=7 WARNING : type of output 2 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11565 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11567 doesn't seem to be define in the database( WARNING : type of input 3 of step 11563 doesn't seem to be define in the database( WARNING : type of output 3 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11568 doesn't seem to be define in the database( WARNING : type of output 1 of step 11568 doesn't seem to be define in the database( WARNING : type of input 3 of step 11566 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11570 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11569 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11570 doesn't seem to be define in the database( WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11569 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11568 have datatype=10 whereas input 3 of step 11571 have datatype=6 WARNING : type of input 2 of step 11573 doesn't seem to be define in the database( WARNING : output 1 of step 11565 have datatype=7 whereas input 2 of step 11573 have datatype=None WARNING : type of output 3 of step 11573 doesn't seem to be define in the database( WARNING : type of input 3 of step 11568 doesn't seem to be define in the database( WARNING : output 0 of step 11568 have datatype=10 whereas input 0 of step 11587 have datatype=18 WARNING : type of input 5 of step 11571 doesn't seem to be define in the database( WARNING : output 0 of step 11587 have datatype=11 whereas input 5 of step 11571 have datatype=None WARNING : output 0 of step 11564 have datatype=1 whereas input 0 of step 11565 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20901273 AND mptpi.`type`=3327 To do Qualite : 0.09945970404899608 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20905767_28-02-2025_03_20_05.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20905767 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20905767 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20902989 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901760 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901761 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20905768 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20905769 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20905782 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20905783 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20902990 order by id desc limit 1 Qualite : 0.0662392077948395 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20905796_28-02-2025_03_23_40.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20905796 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20905796 AND mptpi.`type`=3726 To do Qualite : 0.07982605740529508 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901763_28-02-2025_02_56_19.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901763 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11524 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11533 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11532 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11528 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11528 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11531 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11531 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11525 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11525 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11578 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11578 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11527 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11526 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11526 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11535 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11527 doesn't seem to be define in the database( WARNING : type of input 3 of step 11526 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11524 doesn't seem to be define in the database( WARNING : type of input 2 of step 11528 doesn't seem to be define in the database( WARNING : output 1 of step 11524 have datatype=2 whereas input 1 of step 11531 have datatype=7 WARNING : type of output 2 of step 11531 doesn't seem to be define in the database( WARNING : type of input 1 of step 11525 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11527 have datatype=10 whereas input 3 of step 11534 have datatype=6 WARNING : type of input 2 of step 11578 doesn't seem to be define in the database( WARNING : output 1 of step 11525 have datatype=7 whereas input 2 of step 11578 have datatype=None WARNING : type of output 3 of step 11578 doesn't seem to be define in the database( WARNING : type of input 1 of step 11527 doesn't seem to be define in the database( WARNING : output 0 of step 11527 have datatype=10 whereas input 0 of step 11584 have datatype=18 WARNING : type of input 5 of step 11534 doesn't seem to be define in the database( WARNING : output 0 of step 11584 have datatype=11 whereas input 5 of step 11534 have datatype=None WARNING : type of output 1 of step 11533 doesn't seem to be define in the database( WARNING : type of input 3 of step 11528 doesn't seem to be define in the database( WARNING : type of output 1 of step 11532 doesn't seem to be define in the database( WARNING : type of input 3 of step 11528 doesn't seem to be define in the database( WARNING : output 0 of step 11531 have datatype=1 whereas input 0 of step 11525 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20901763 AND mptpi.`type`=4211 To do Qualite : 0.1992153555709031 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901764_28-02-2025_02_59_22.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901764 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11560 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11567 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11567 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11563 crop_condition is not consistent : 4 used against 2 in the step definition ! Step 11563 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11564 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11564 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11573 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11573 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11566 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11566 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 1 of step 11560 have datatype=2 whereas input 1 of step 11564 have datatype=7 WARNING : type of output 2 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11565 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11567 doesn't seem to be define in the database( WARNING : type of input 3 of step 11563 doesn't seem to be define in the database( WARNING : type of output 3 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11568 doesn't seem to be define in the database( WARNING : type of output 1 of step 11568 doesn't seem to be define in the database( WARNING : type of input 3 of step 11566 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11570 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11569 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11570 doesn't seem to be define in the database( WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11569 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11568 have datatype=10 whereas input 3 of step 11571 have datatype=6 WARNING : type of input 2 of step 11573 doesn't seem to be define in the database( WARNING : output 1 of step 11565 have datatype=7 whereas input 2 of step 11573 have datatype=None WARNING : type of output 3 of step 11573 doesn't seem to be define in the database( WARNING : type of input 3 of step 11568 doesn't seem to be define in the database( WARNING : output 0 of step 11568 have datatype=10 whereas input 0 of step 11587 have datatype=18 WARNING : type of input 5 of step 11571 doesn't seem to be define in the database( WARNING : output 0 of step 11587 have datatype=11 whereas input 5 of step 11571 have datatype=None WARNING : output 0 of step 11564 have datatype=1 whereas input 0 of step 11565 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20901764 AND mptpi.`type`=3327 To do Qualite : 0.19853583693421006 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901765_28-02-2025_03_19_59.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901765 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11560 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11567 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11567 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11563 crop_condition is not consistent : 4 used against 2 in the step definition ! Step 11563 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11564 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11564 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11573 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11573 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11566 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11566 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 1 of step 11560 have datatype=2 whereas input 1 of step 11564 have datatype=7 WARNING : type of output 2 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11565 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11567 doesn't seem to be define in the database( WARNING : type of input 3 of step 11563 doesn't seem to be define in the database( WARNING : type of output 3 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11568 doesn't seem to be define in the database( WARNING : type of output 1 of step 11568 doesn't seem to be define in the database( WARNING : type of input 3 of step 11566 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11570 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11569 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11570 doesn't seem to be define in the database( WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11569 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11568 have datatype=10 whereas input 3 of step 11571 have datatype=6 WARNING : type of input 2 of step 11573 doesn't seem to be define in the database( WARNING : output 1 of step 11565 have datatype=7 whereas input 2 of step 11573 have datatype=None WARNING : type of output 3 of step 11573 doesn't seem to be define in the database( WARNING : type of input 3 of step 11568 doesn't seem to be define in the database( WARNING : output 0 of step 11568 have datatype=10 whereas input 0 of step 11587 have datatype=18 WARNING : type of input 5 of step 11571 doesn't seem to be define in the database( WARNING : output 0 of step 11587 have datatype=11 whereas input 5 of step 11571 have datatype=None WARNING : output 0 of step 11564 have datatype=1 whereas input 0 of step 11565 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20901765 AND mptpi.`type`=3327 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901276 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901766 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901278 order by id desc limit 1 Qualite : 0.191975621870083 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901279_28-02-2025_03_04_48.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901279 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20901279 AND mptpi.`type`=3594 To do find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901767_28-02-2025_01_59_06.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901767 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901768 order by id desc limit 1 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901769_28-02-2025_01_39_56.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901769 order by id desc limit 1 Qualite : 0.08650521466694104 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20902992_28-02-2025_03_01_07.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20902992 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20902992 AND mptpi.`type`=4207 To do Qualite : 0.0950331596087939 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20905797_28-02-2025_03_18_35.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20905797 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20905797 AND mptpi.`type`=4207 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901283 order by id desc limit 1 Qualite : 0.1199733468175425 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901284_28-02-2025_01_54_57.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901284 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20901284 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901285 order by id desc limit 1 Qualite : 0.16412012537548684 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901286_28-02-2025_02_31_23.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901286 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20901286 AND mptpi.`type`=3594 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901287 order by id desc limit 1 Qualite : 0.21264894399468545 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901288_28-02-2025_02_10_03.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901288 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20901288 AND mptpi.`type`=3594 To do Qualite : 0.0662862777318847 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901301_28-02-2025_00_57_08.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901301 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20901301 AND mptpi.`type`=4207 To do Qualite : 0.10326228661809093 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901314_28-02-2025_01_38_23.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901314 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20901314 AND mptpi.`type`=4207 To do Qualite : 0.059403225739113 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20901315_28-02-2025_01_08_40.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20901315 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20901315 AND mptpi.`type`=4207 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'27022025': {'nb_upload': 20, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1340018829, 1340018826, 1340018786, 1340018755, 1339929112, 1339928413, 1339928409, 1339928405, 1339928400, 1339928322] Looping around the photos to save general results len do output : 1 /20905796Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018829', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018826', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018786', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1340018755', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339929112', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928413', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928409', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928405', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928400', None, None, None, None, None, '2621865') ('3459', None, None, None, None, None, None, None, '2621865') ('3459', None, '1339928322', None, None, None, None, None, '2621865') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.10030961036682129 save_final save missing photos in datou_result : time spend for datou_step_exec : 17.24962329864502 time spend to save output : 0.10072541236877441 total time spend for step 11 : 17.350348711013794 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 125.59user 151.31system 5:50.61elapsed 78%CPU (0avgtext+0avgdata 4127784maxresident)k 2611224inputs+132480outputs (58502major+11814067minor)pagefaults 0swaps