python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 20 -a ' -a 4189' -s datou_current_4189 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 1152221 load datou : 4189 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 list_input_json: {} Current got : datou_id : 4189, datou_cur_ids : ['2727057'] with mtr_portfolio_ids : ['22064633'] and first list_photo_ids : [] new path : /proc/1152221/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : split_time_score over limit max, limiting to limit_max 100 list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB time to download the photos : 0.03147435188293457 About to test input to load Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 1 step1:split_time_score Fri Apr 4 16:00:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec begin split time score 2022-04-13 10:29:59 0 TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 3379, 'mtr_user_id': 31, 'name': 'learn_classif_flux_maj_generique_effnet_v2_s_02062022', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'aluminium,ela,film_pedb,flux_dev,jrm,pcm,pcnc,pehd_pp,pet_clair,refus,tapis_vide', 'svm_portfolios_learning': '5515864,5515840,5515844,5515850,6244400,6237996,6237998,5515847,5515841,5515868,5515866', 'photo_hashtag_type': 4374, 'photo_desc_type': 5680, 'type_classification': 'tf_classification2', 'hashtag_id_list': '493546845,492741797,2107760237,2107760238,495916461,560181804,1284539308,2107760239,2107755846,538914404,2107748999'}] thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('10', 359), ('14', 356)) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {1: 712} 04042025 22064633 Nombre de photos uploadées : 715 / 23040 (3%) 04042025 22064633 Nombre de photos taguées (types de déchets): 712 / 715 (99%) 04042025 22064633 Nombre de photos taguées (volume) : 712 / 715 (99%) elapsed_time : load_data_split_time_score 3.5762786865234375e-06 elapsed_time : order_list_meta_photo_and_scores 0.0003044605255126953 LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL??? elapsed_time : fill_and_build_computed_from_old_data 0.06243562698364258 elapsed_time : insert_dashboard_record_day_entry 0.031187772750854492 LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL Creating list_photo_by_hashtags Hashtag is None Hashtag is None Hashtag is None elapsed_time : list_photo_by_hashtags 0.03774881362915039 Creating list_photo_total elapsed_time : select_descriptors 26.262598037719727 04042025 22064633 Nombre de photos avec descriptors (type 5680) : 347 / 350 (99%) ERROR : Hum hum, what can we do for different size of descriptors (ignore the difference ) : 0 vs 1280 photo_id : 1350018596 photo_id_prec : 0 0:00:00|ON:LL0:04:59.990452|OFF:LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL0:10:59.275735|ON:LLL0:29:59.789471|OFF:LLLLLLLLLL0:01:00.821364|ON:LL0:03:59.201814|OFF:0:00:00|ON:LMissing descriptors for photos 1350032529 and 1350032627 LMissing descriptors for photos 1350032627 and 1350032690 LMissing descriptors for photos 1350032690 and 1350033088 LMissing descriptors for photos 1350033088 and 1350033163 LMissing descriptors for photos 1350033163 and 1350033192 LMissing descriptors for photos 1350033192 and 1350033299 LMissing descriptors for photos 1350033299 and 1350033425 LLMissing descriptors for photos 1350033881 and 1350034193 LLLL0:08:00.189707|OFF:L3:00:59.945580|ON:Missing descriptors for photos 1350091701 and 1350091778 LMissing descriptors for photos 1350091778 and 1350091839 LMissing descriptors for photos 1350091839 and 1350091879 LMissing descriptors for photos 1350091879 and 1350091895 LMissing descriptors for photos 1350091895 and 1350092111 LMissing descriptors for photos 1350092111 and 1350092166 LMissing descriptors for photos 1350092166 and 1350092208 LMissing descriptors for photos 1350092208 and 1350092269 LMissing descriptors for photos 1350092269 and 1350092292 LMissing descriptors for photos 1350092292 and 1350092317 LMissing descriptors for photos 1350092317 and 1350092624 LMissing descriptors for photos 1350092624 and 1350092658 LMissing descriptors for photos 1350092658 and 1350092708 LMissing descriptors for photos 1350092708 and 1350092751 LMissing descriptors for photos 1350092751 and 1350092882 LMissing descriptors for photos 1350092882 and 1350093016 LMissing descriptors for photos 1350093016 and 1350093280 LMissing descriptors for photos 1350093280 and 1350093320 LMissing descriptors for photos 1350093320 and 1350093373 LMissing descriptors for photos 1350093373 and 1350093403 LMissing descriptors for photos 1350093403 and 1350093440 LMissing descriptors for photos 1350093440 and 1350093488 LMissing descriptors for photos 1350093488 and 1350093734 LMissing descriptors for photos 1350093734 and 1350093778 LMissing descriptors for photos 1350093778 and 1350093805 LMissing descriptors for photos 1350093805 and 1350093834 LMissing descriptors for photos 1350093834 and 1350093872 LMissing descriptors for photos 1350093872 and 1350093899 LMissing descriptors for photos 1350093899 and 1350093978 LMissing descriptors for photos 1350093978 and 1350094041 LMissing descriptors for photos 1350094041 and 1350094045 LMissing descriptors for photos 1350094045 and 1350094086 LMissing descriptors for photos 1350094086 and 1350094096 LMissing descriptors for photos 1350094096 and 1350094104 LMissing descriptors for photos 1350094104 and 1350094267 LMissing descriptors for photos 1350094267 and 1350094320 LMissing descriptors for photos 1350094320 and 1350094392 LMissing descriptors for photos 1350094392 and 1350094495 LMissing descriptors for photos 1350094495 and 1350094563 LMissing descriptors for photos 1350094563 and 1350094622 LMissing descriptors for photos 1350094622 and 1350094816 LMissing descriptors for photos 1350094816 and 1350094866 LMissing descriptors for photos 1350094866 and 1350094874 LMissing descriptors for photos 1350094874 and 1350094917 LMissing descriptors for photos 1350094917 and 1350094933 LMissing descriptors for photos 1350094933 and 1350094947 LMissing descriptors for photos 1350094947 and 1350095161 LMissing descriptors for photos 1350095161 and 1350095181 LMissing descriptors for photos 1350095181 and 1350095196 LMissing descriptors for photos 1350095196 and 1350095206 LMissing descriptors for photos 1350095206 and 1350095242 LMissing descriptors for photos 1350095242 and 1350095273 LMissing descriptors for photos 1350095273 and 1350095316 LMissing descriptors for photos 1350095316 and 1350095324 LMissing descriptors for photos 1350095324 and 1350095335 LMissing descriptors for photos 1350095335 and 1350095341 LMissing descriptors for photos 1350095341 and 1350095354 LMissing descriptors for photos 1350095354 and 1350095364 LMissing descriptors for photos 1350095364 and 1350095472 LMissing descriptors for photos 1350095472 and 1350095522 LMissing descriptors for photos 1350095522 and 1350095527 LMissing descriptors for photos 1350095527 and 1350095537 LMissing descriptors for photos 1350095537 and 1350095543 LMissing descriptors for photos 1350095543 and 1350095577 LMissing descriptors for photos 1350095577 and 1350095620 LMissing descriptors for photos 1350095620 and 1350095691 LMissing descriptors for photos 1350095691 and 1350095815 LMissing descriptors for photos 1350095815 and 1350095874 LMissing descriptors for photos 1350095874 and 1350095937 LMissing descriptors for photos 1350095937 and 1350095981 LMissing descriptors for photos 1350095981 and 1350096382 LMissing descriptors for photos 1350096382 and 1350096488 LMissing descriptors for photos 1350096488 and 1350096578 LMissing descriptors for photos 1350096578 and 1350096586 LMissing descriptors for photos 1350096586 and 1350096596 LMissing descriptors for photos 1350096596 and 1350096600 LMissing descriptors for photos 1350096600 and 1350096715 LMissing descriptors for photos 1350096715 and 1350096726 LMissing descriptors for photos 1350096726 and 1350096738 LMissing descriptors for photos 1350096738 and 1350096787 LMissing descriptors for photos 1350096787 and 1350096806 LMissing descriptors for photos 1350096806 and 1350096849 LMissing descriptors for photos 1350096849 and 1350097033 LMissing descriptors for photos 1350097033 and 1350097066 LMissing descriptors for photos 1350097066 and 1350097097 LMissing descriptors for photos 1350097097 and 1350097145 LMissing descriptors for photos 1350097145 and 1350097184 LMissing descriptors for photos 1350097184 and 1350097206 LMissing descriptors for photos 1350097206 and 1350097247 LMissing descriptors for photos 1350097247 and 1350097255 LMissing descriptors for photos 1350097255 and 1350097260 LMissing descriptors for photos 1350097260 and 1350097265 LMissing descriptors for photos 1350097265 and 1350097271 LMissing descriptors for photos 1350097271 and 1350097286 LMissing descriptors for photos 1350097286 and 1350097413 LMissing descriptors for photos 1350097413 and 1350097424 LMissing descriptors for photos 1350097424 and 1350097442 LMissing descriptors for photos 1350097442 and 1350097451 LMissing descriptors for photos 1350097451 and 1350097494 LMissing descriptors for photos 1350097494 and 1350097516 LMissing descriptors for photos 1350097516 and 1350097566 LMissing descriptors for photos 1350097566 and 1350097578 LMissing descriptors for photos 1350097578 and 1350097586 LMissing descriptors for photos 1350097586 and 1350097614 LMissing descriptors for photos 1350097614 and 1350097684 LMissing descriptors for photos 1350097684 and 1350097689 LMissing descriptors for photos 1350097689 and 1350097762 LMissing descriptors for photos 1350097762 and 1350097771 LMissing descriptors for photos 1350097771 and 1350097795 LMissing descriptors for photos 1350097795 and 1350097800 LMissing descriptors for photos 1350097800 and 1350097816 LMissing descriptors for photos 1350097816 and 1350097825 LMissing descriptors for photos 1350097825 and 1350097914 LMissing descriptors for photos 1350097914 and 1350097933 LMissing descriptors for photos 1350097933 and 1350097941 LMissing descriptors for photos 1350097941 and 1350097948 LMissing descriptors for photos 1350097948 and 1350097954 LMissing descriptors for photos 1350097954 and 1350097968 LMissing descriptors for photos 1350097968 and 1350098005 LMissing descriptors for photos 1350098005 and 1350098012 LMissing descriptors for photos 1350098012 and 1350098039 LMissing descriptors for photos 1350098039 and 1350098063 LMissing descriptors for photos 1350098063 and 1350098073 LMissing descriptors for photos 1350098073 and 1350098083 LMissing descriptors for photos 1350098083 and 1350098411 LMissing descriptors for photos 1350098411 and 1350098446 LMissing descriptors for photos 1350098446 and 1350098464 LMissing descriptors for photos 1350098464 and 1350098470 LMissing descriptors for photos 1350098470 and 1350098480 LMissing descriptors for photos 1350098480 and 1350098487 LMissing descriptors for photos 1350098487 and 1350098533 LMissing descriptors for photos 1350098533 and 1350098539 LMissing descriptors for photos 1350098539 and 1350098546 LMissing descriptors for photos 1350098546 and 1350098555 LMissing descriptors for photos 1350098555 and 1350098644 LMissing descriptors for photos 1350098644 and 1350098681 LMissing descriptors for photos 1350098681 and 1350098773 LMissing descriptors for photos 1350098773 and 1350098776 LMissing descriptors for photos 1350098776 and 1350098787 LMissing descriptors for photos 1350098787 and 1350098813 LMissing descriptors for photos 1350098813 and 1350098819 LMissing descriptors for photos 1350098819 and 1350098828 LMissing descriptors for photos 1350098828 and 1350098911 LMissing descriptors for photos 1350098911 and 1350098935 LMissing descriptors for photos 1350098935 and 1350098978 LMissing descriptors for photos 1350098978 and 1350099018 LMissing descriptors for photos 1350099018 and 1350099060 LMissing descriptors for photos 1350099060 and 1350099114 LMissing descriptors for photos 1350099114 and 1350099419 LMissing descriptors for photos 1350099419 and 1350099502 LMissing descriptors for photos 1350099502 and 1350099556 LMissing descriptors for photos 1350099556 and 1350099604 LMissing descriptors for photos 1350099604 and 1350099651 LMissing descriptors for photos 1350099651 and 1350099702 LMissing descriptors for photos 1350099702 and 1350099899 LMissing descriptors for photos 1350099899 and 1350099945 LMissing descriptors for photos 1350099945 and 1350099956 LMissing descriptors for photos 1350099956 and 1350099986 LMissing descriptors for photos 1350099986 and 1350100001 LMissing descriptors for photos 1350100001 and 1350100009 LMissing descriptors for photos 1350100009 and 1350100093 LMissing descriptors for photos 1350100093 and 1350100108 LMissing descriptors for photos 1350100108 and 1350100155 LMissing descriptors for photos 1350100155 and 1350100198 LMissing descriptors for photos 1350100198 and 1350100226 LMissing descriptors for photos 1350100226 and 1350100261 LMissing descriptors for photos 1350100261 and 1350100346 LMissing descriptors for photos 1350100346 and 1350100357 LMissing descriptors for photos 1350100357 and 1350100371 LMissing descriptors for photos 1350100371 and 1350100386 LMissing descriptors for photos 1350100386 and 1350100395 LMissing descriptors for photos 1350100395 and 1350100436 LMissing descriptors for photos 1350100436 and 1350100487 LMissing descriptors for photos 1350100487 and 1350100515 LMissing descriptors for photos 1350100515 and 1350100590 LMissing descriptors for photos 1350100590 and 1350100601 LMissing descriptors for photos 1350100601 and 1350100610 LMissing descriptors for photos 1350100610 and 1350100622 LMissing descriptors for photos 1350100622 and 1350100894 LMissing descriptors for photos 1350100894 and 1350100920 LMissing descriptors for photos 1350100920 and 1350100933 LMissing descriptors for photos 1350100933 and 1350100941 LMissing descriptors for photos 1350100941 and 1350100949 LMissing descriptors for photos 1350100949 and 1350100961 LMissing descriptors for photos 1350100961 and 1350101056 LMissing descriptors for photos 1350101056 and 1350101178 LMissing descriptors for photos 1350101178 and 1350101343 LMissing descriptors for photos 1350101343 and 1350101434 LMissing descriptors for photos 1350101434 and 1350101490 LMissing descriptors for photos 1350101490 and 1350101498 LMissing descriptors for photos 1350101498 and 1350101571 LMissing descriptors for photos 1350101571 and 1350101593 LMissing descriptors for photos 1350101593 and 1350101599 LMissing descriptors for photos 1350101599 and 1350101622 LMissing descriptors for photos 1350101622 and 1350101646 LMissing descriptors for photos 1350101646 and 1350101681 LMissing descriptors for photos 1350101681 and 1350101837 LMissing descriptors for photos 1350101837 and 1350101845 LMissing descriptors for photos 1350101845 and 1350101886 LMissing descriptors for photos 1350101886 and 1350101931 LMissing descriptors for photos 1350101931 and 1350102009 LMissing descriptors for photos 1350102009 and 1350102066 LMissing descriptors for photos 1350102066 and 1350102189 LMissing descriptors for photos 1350102189 and 1350102198 LMissing descriptors for photos 1350102198 and 1350102238 LMissing descriptors for photos 1350102238 and 1350102267 LMissing descriptors for photos 1350102267 and 1350102310 LMissing descriptors for photos 1350102310 and 1350102359 LMissing descriptors for photos 1350102359 and 1350102476 LMissing descriptors for photos 1350102476 and 1350102499 LMissing descriptors for photos 1350102499 and 1350102534 LMissing descriptors for photos 1350102534 and 1350102589 LMissing descriptors for photos 1350102589 and 1350102639 LMissing descriptors for photos 1350102639 and 1350102645 LMissing descriptors for photos 1350102645 and 1350102782 LMissing descriptors for photos 1350102782 and 1350102788 LMissing descriptors for photos 1350102788 and 1350102795 LMissing descriptors for photos 1350102795 and 1350102810 LMissing descriptors for photos 1350102810 and 1350102882 LMissing descriptors for photos 1350102882 and 1350102886 LMissing descriptors for photos 1350102886 and 1350103013 LMissing descriptors for photos 1350103013 and 1350103051 LMissing descriptors for photos 1350103051 and 1350103095 LMissing descriptors for photos 1350103095 and 1350103101 LMissing descriptors for photos 1350103101 and 1350103108 LMissing descriptors for photos 1350103108 and 1350103115 LMissing descriptors for photos 1350103115 and 1350103157 LMissing descriptors for photos 1350103157 and 1350103168 LMissing descriptors for photos 1350103168 and 1350103177 LMissing descriptors for photos 1350103177 and 1350103188 LMissing descriptors for photos 1350103188 and 1350103198 LMissing descriptors for photos 1350103198 and 1350103202 LMissing descriptors for photos 1350103202 and 1350103334 LMissing descriptors for photos 1350103334 and 1350103349 LMissing descriptors for photos 1350103349 and 1350103359 LMissing descriptors for photos 1350103359 and 1350103413 LMissing descriptors for photos 1350103413 and 1350103455 LMissing descriptors for photos 1350103455 and 1350103488 LMissing descriptors for photos 1350103488 and 1350103553 LMissing descriptors for photos 1350103553 and 1350103563 LMissing descriptors for photos 1350103563 and 1350103581 LMissing descriptors for photos 1350103581 and 1350103586 LMissing descriptors for photos 1350103586 and 1350103592 LMissing descriptors for photos 1350103592 and 1350103613 LMissing descriptors for photos 1350103613 and 1350103691 LMissing descriptors for photos 1350103691 and 1350103696 LMissing descriptors for photos 1350103696 and 1350103704 LMissing descriptors for photos 1350103704 and 1350103721 LMissing descriptors for photos 1350103721 and 1350103727 LMissing descriptors for photos 1350103727 and 1350103740 LMissing descriptors for photos 1350103740 and 1350103874 LMissing descriptors for photos 1350103874 and 1350103892 LMissing descriptors for photos 1350103892 and 1350103930 LMissing descriptors for photos 1350103930 and 1350103968 LMissing descriptors for photos 1350103968 and 1350104005 LMissing descriptors for photos 1350104005 and 1350104013 LMissing descriptors for photos 1350104013 and 1350104061 LMissing descriptors for photos 1350104061 and 1350104064 LMissing descriptors for photos 1350104064 and 1350104068 LMissing descriptors for photos 1350104068 and 1350104076 LMissing descriptors for photos 1350104076 and 1350104086 LMissing descriptors for photos 1350104086 and 1350104091 LMissing descriptors for photos 1350104091 and 1350104139 LMissing descriptors for photos 1350104139 and 1350104144 LMissing descriptors for photos 1350104144 and 1350104148 LMissing descriptors for photos 1350104148 and 1350104155 LMissing descriptors for photos 1350104155 and 1350104204 LMissing descriptors for photos 1350104204 and 1350104212 LMissing descriptors for photos 1350104212 and 1350104285 LMissing descriptors for photos 1350104285 and 1350104299 LMissing descriptors for photos 1350104299 and 1350104315 LMissing descriptors for photos 1350104315 and 1350104324 LMissing descriptors for photos 1350104324 and 1350104399 LMissing descriptors for photos 1350104399 and 1350104536 LMissing descriptors for photos 1350104536 and 1350104788 LMissing descriptors for photos 1350104788 and 1350104859 LMissing descriptors for photos 1350104859 and 1350104875 LMissing descriptors for photos 1350104875 and 1350104898 LMissing descriptors for photos 1350104898 and 1350104913 LMissing descriptors for photos 1350104913 and 1350104926 LMissing descriptors for photos 1350104926 and 1350104968 LMissing descriptors for photos 1350104968 and 1350104977 LMissing descriptors for photos 1350104977 and 1350105002 LMissing descriptors for photos 1350105002 and 1350105047 LMissing descriptors for photos 1350105047 and 1350105087 LMissing descriptors for photos 1350105087 and 1350105123 LMissing descriptors for photos 1350105123 and 1350105151 LMissing descriptors for photos 1350105151 and 1350105167 LMissing descriptors for photos 1350105167 and 1350105174 LMissing descriptors for photos 1350105174 and 1350105182 LMissing descriptors for photos 1350105182 and 1350105185 LMissing descriptors for photos 1350105185 and 1350105192 LMissing descriptors for photos 1350105192 and 1350105231 LMissing descriptors for photos 1350105231 and 1350105239 LMissing descriptors for photos 1350105239 and 1350105246 LMissing descriptors for photos 1350105246 and 1350105268 LMissing descriptors for photos 1350105268 and 1350105274 LMissing descriptors for photos 1350105274 and 1350105280 LMissing descriptors for photos 1350105280 and 1350105319 LMissing descriptors for photos 1350105319 and 1350105352 LMissing descriptors for photos 1350105352 and 1350105403 LMissing descriptors for photos 1350105403 and 1350105431 LMissing descriptors for photos 1350105431 and 1350105473 LMissing descriptors for photos 1350105473 and 1350105529 LMissing descriptors for photos 1350105529 and 1350106307 LMissing descriptors for photos 1350106307 and 1350106388 LMissing descriptors for photos 1350106388 and 1350106518 LMissing descriptors for photos 1350106518 and 1350106685 LMissing descriptors for photos 1350106685 and 1350106792 LMissing descriptors for photos 1350106792 and 1350106857 LMissing descriptors for photos 1350106857 and 1350107354 LMissing descriptors for photos 1350107354 and 1350107438 LMissing descriptors for photos 1350107438 and 1350107515 LMissing descriptors for photos 1350107515 and 1350107610 LMissing descriptors for photos 1350107610 and 1350107682 LMissing descriptors for photos 1350107682 and 1350107828 LMissing descriptors for photos 1350107828 and 1350108133 LMissing descriptors for photos 1350108133 and 1350108184 LMissing descriptors for photos 1350108184 and 1350108214 LMissing descriptors for photos 1350108214 and 1350108222 LMissing descriptors for photos 1350108222 and 1350108317 LMissing descriptors for photos 1350108317 and 1350108332 LMissing descriptors for photos 1350108332 and 1350108571 LMissing descriptors for photos 1350108571 and 1350108642 LMissing descriptors for photos 1350108642 and 1350108717 LMissing descriptors for photos 1350108717 and 1350108801 LMissing descriptors for photos 1350108801 and 1350108958 LMissing descriptors for photos 1350108958 and 1350109058 LMissing descriptors for photos 1350109058 and 1350109214 LMissing descriptors for photos 1350109214 and 1350109336 LMissing descriptors for photos 1350109336 and 1350109432 LMissing descriptors for photos 1350109432 and 1350109475 LMissing descriptors for photos 1350109475 and 1350109511 LMissing descriptors for photos 1350109511 and 1350109521 LMissing descriptors for photos 1350109521 and 1350109635 LMissing descriptors for photos 1350109635 and 1350109644 LMissing descriptors for photos 1350109644 and 1350109653 LMissing descriptors for photos 1350109653 and 1350109667 LMissing descriptors for photos 1350109667 and 1350109687 LMissing descriptors for photos 1350109687 and 1350109702 LMissing descriptors for photos 1350109702 and 1350109852 LMissing descriptors for photos 1350109852 and 1350109865 LMissing descriptors for photos 1350109865 and 1350109868 LMissing descriptors for photos 1350109868 and 1350109894 LMissing descriptors for photos 1350109894 and 1350109924 LMissing descriptors for photos 1350109924 and 1350109935 LMissing descriptors for photos 1350109935 and 1350110041 LMissing descriptors for photos 1350110041 and 1350110086 LMissing descriptors for photos 1350110086 and 1350110100 LMissing descriptors for photos 1350110100 and 1350110130 LMissing descriptors for photos 1350110130 and 1350110149 LMissing descriptors for photos 1350110149 and 1350110175 LMissing descriptors for photos 1350110175 and 1350110224 LMissing descriptors for photos 1350110224 and 1350110256 LMissing descriptors for photos 1350110256 and 1350110268 L 04042025 Removing 75 photos because of the 'same image' condition Total on : 2819.171444 list_time_on Total off : 11580.042679 list_time_off dist_desc begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 715 time used for this insertion : 0.12696337699890137 photos_removed : len 75 elapsed_time : remove_photo_duplicate 0.19369149208068848 To do, maybe not at the correct place ! ...................................L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L..................................................................L..............................................................................................................L.L.L.L.L.L.........................L..............L.L.L.L.L.L.L.L..L.L..L..........................L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.Lforce hashtag to JRM elapsed_time : CREATE_PORT_BATCH_BY_HOUR 0.02308344841003418 NUMBER BATCH : 5 list_ponderation used : [1e-05, 1e-05, 1e-05, 1e-05, 1e-05] , list_hashtag_class_create_as_list : ['jrm'] result_one_balle_Type_JRM:{'day': '04042025', 'map_nb_amount': {0: 9, 1: 24, 2: 0, 3: 1, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 299.9904520511627, 'nb_balles_papier': 0.00034000000000000024, 'begin_time_port': 'image_04042025_10_00_03_241952m0.jpg 1e-05 for time 1, id_amount 1 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 0.00034000000000000024 result_one_balle_Type_JRM:{'day': '04042025', 'map_nb_amount': {0: 12, 1: 64, 2: 2, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 779.6991782188416, 'nb_balles_papier': 0.0007800000000000014, 'begin_time_port': 'image_04042025_10_16_02_508139m0.jpg 1e-05 for time 1, id_amount 2 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 0.0007800000000000014 result_one_balle_Type_JRM:{'day': '04042025', 'map_nb_amount': {0: 10, 1: 78, 2: 0, 3: 2, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 839.7666509151459, 'nb_balles_papier': 0.0009000000000000017, 'begin_time_port': 'image_04042025_10_30_02_935687m0.jpg 1e-05 for time 1, id_amount 1 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 0.0009000000000000017 result_one_balle_Type_JRM:{'day': '04042025', 'map_nb_amount': {0: 9, 1: 60, 2: 0, 3: 1, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 840.0107159614563, 'nb_balles_papier': 0.0007000000000000012, 'begin_time_port': 'image_04042025_10_45_02_499779m0.jpg 1e-05 for time 1, id_amount 2 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 0.0007000000000000012 We have rejected 3 photos because of the batch_size condition ! NUMBER BATCH list_of_portfolios_to_create : 4 list_same_port_ids : [22065377] find same portfolio which already exist 22065377 , we will use it list_same_port_ids : [22066033] find same portfolio which already exist 22066033 , we will use it list_same_port_ids : [22067422] find same portfolio which already exist 22067422 , we will use it list_same_port_ids : [22070699] find same portfolio which already exist 22070699 , we will use it # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=22065377 AND mptpi.`type`=3726 To do # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=22066033 AND mptpi.`type`=3726 To do # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=22067422 AND mptpi.`type`=3726 To do # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=22070699 AND mptpi.`type`=3726 To do elapsed_time : count_nb_balles_and_create_portfolio 1.971942663192749 # DISPLAY ALL COLLECTED DATA : {'04042025': {'nb_upload': 715, 'nb_taggue_class': 712, 'nb_taggue_densite': 712, 'nb_descriptors': 347}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1350110268, 1350110256, 1350110224, 1350110175, 1350110149, 1350110130, 1350110100, 1350110086, 1350110041, 1350109935, 1350109924, 1350109894, 1350109868, 1350109865, 1350109852, 1350109702, 1350109687, 1350109667, 1350109653, 1350109644, 1350109635, 1350109521, 1350109511, 1350109475, 1350109432, 1350109336, 1350109214, 1350109058, 1350108958, 1350108801, 1350108717, 1350108642, 1350108571, 1350108332, 1350108317, 1350108222, 1350108214, 1350108184, 1350108133, 1350107828, 1350107682, 1350107610, 1350107515, 1350107438, 1350107354, 1350106857, 1350106792, 1350106685, 1350106518, 1350106388, 1350106307, 1350105529, 1350105473, 1350105431, 1350105403, 1350105352, 1350105319, 1350105280, 1350105274, 1350105268, 1350105246, 1350105239, 1350105231, 1350105192, 1350105185, 1350105182, 1350105174, 1350105167, 1350105151, 1350105123, 1350105087, 1350105047, 1350105002, 1350104977, 1350104968, 1350104926, 1350104913, 1350104898, 1350104875, 1350104859, 1350104788, 1350104536, 1350104399, 1350104324, 1350104315, 1350104299, 1350104285, 1350104212, 1350104204, 1350104155, 1350104148, 1350104144, 1350104139, 1350104091, 1350104086, 1350104076, 1350104068, 1350104064, 1350104061, 1350104013] Looping around the photos to save general results len do output : 1 /22064633Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350110268', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350110256', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350110224', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350110175', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350110149', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350110130', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350110100', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350110086', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350110041', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350109935', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350109924', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350109894', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350109868', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350109865', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350109852', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350109702', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350109687', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350109667', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350109653', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350109644', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350109635', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350109521', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350109511', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350109475', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350109432', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350109336', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350109214', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350109058', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350108958', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350108801', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350108717', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350108642', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350108571', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350108332', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350108317', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350108222', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350108214', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350108184', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350108133', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350107828', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350107682', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350107610', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350107515', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350107438', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350107354', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350106857', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350106792', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350106685', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350106518', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350106388', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350106307', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350105529', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350105473', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350105431', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350105403', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350105352', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350105319', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350105280', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350105274', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350105268', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350105246', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350105239', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350105231', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350105192', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350105185', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350105182', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350105174', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350105167', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350105151', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350105123', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350105087', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350105047', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350105002', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350104977', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350104968', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350104926', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350104913', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350104898', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350104875', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350104859', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350104788', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350104536', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350104399', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350104324', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350104315', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350104299', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350104285', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350104212', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350104204', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350104155', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350104148', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350104144', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350104139', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350104091', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350104086', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350104076', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350104068', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350104064', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350104061', None, None, None, None, None, '2727057') ('4189', None, None, None, None, None, None, None, '2727057') ('4189', '22064633', '1350104013', None, None, None, None, None, '2727057') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 101 time used for this insertion : 0.03173494338989258 save_final save missing photos in datou_result : time spend for datou_step_exec : 28.72433829307556 time spend to save output : 0.032652854919433594 total time spend for step 1 : 28.756991147994995 caffe_path_current : About to save ! 2 After save, about to update current ! ret : 2 len(input) + len(total_photo_id_missing) : 1 set_done_treatment 2.77user 0.91system 0:32.89elapsed 11%CPU (0avgtext+0avgdata 107784maxresident)k 352inputs+2936outputs (0major+50213minor)pagefaults 0swaps