python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 20 -a ' -a 4189' -s datou_current_4189 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 768884 load datou : 4189 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 list_input_json: [] Current got : datou_id : 4189, datou_cur_ids : ['4169169'] with mtr_portfolio_ids : ['28772302'] and first list_photo_ids : [] new path : /proc/768884/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : split_time_score over limit max, limiting to limit_max 100 list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB time to download the photos : 0.09583926200866699 About to test input to load Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 1 step1:split_time_score Tue Dec 2 09:05:27 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec begin split time score 2022-04-13 10:29:59 0 TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 3379, 'mtr_user_id': 31, 'name': 'learn_classif_flux_maj_generique_effnet_v2_s_02062022', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'aluminium,ela,film_pedb,flux_dev,jrm,pcm,pcnc,pehd_pp,pet_clair,refus,tapis_vide', 'svm_portfolios_learning': '5515864,5515840,5515844,5515850,6244400,6237996,6237998,5515847,5515841,5515868,5515866', 'photo_hashtag_type': 4374, 'photo_desc_type': 5680, 'type_classification': 'tf_classification2', 'hashtag_id_list': '493546845,492741797,2107760237,2107760238,495916461,560181804,1284539308,2107760239,2107755846,538914404,2107748999'}] thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('10', 373), ('14', 355)) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {1: 728} 21112025 28772302 Nombre de photos uploadées : 728 / 23040 (3%) 21112025 28772302 Nombre de photos taguées (types de déchets): 728 / 728 (100%) 21112025 28772302 Nombre de photos taguées (volume) : 728 / 728 (100%) elapsed_time : load_data_split_time_score 4.291534423828125e-06 elapsed_time : order_list_meta_photo_and_scores 0.0003032684326171875 LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL elapsed_time : fill_and_build_computed_from_old_data 0.03727602958679199 Catched exception ! Connect or reconnect ! Catched exception ! Connect or reconnect ! elapsed_time : insert_dashboard_record_day_entry 0.24086689949035645 LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL Creating list_photo_by_hashtags elapsed_time : list_photo_by_hashtags 0.04684782028198242 Creating list_photo_total elapsed_time : select_descriptors 31.14802885055542 21112025 28772302 Nombre de photos avec descriptors (type 5680) : 373 / 373 (100%) ERROR : Hum hum, what can we do for different size of descriptors (ignore the difference ) : 0 vs 1280 photo_id : 1395475128 photo_id_prec : 0 0:00:00|ON:0:08:00.368952|OFF:0:00:00|ON:LL0:02:59.191232|OFF:0:00:00|ON:LL0:16:00.577121|OFF:LLLLLLLLLLLLLLLLLLLLLL0:02:00.233526|ON:LL0:00:59.815016|OFF:0:00:00|ON:LL0:00:00|OFF:LLLLLLLLLLLLLLLLLL0:01:59.773415|ON:LL0:00:59.855565|OFF:0:00:00|ON:LL0:06:59.741523|OFF:0:00:00|ON:LL0:03:00.160803|OFF:0:00:00|ON:LL0:00:59.699031|OFF:LLLL0:00:00|ON:LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL0:14:00.051223|OFF:L0:00:00|ON:LLLLLLLLMissing descriptors for photos 1397285159 and 1397285161 LMissing descriptors for photos 1397285161 and 1397285166 LMissing descriptors for photos 1397285166 and 1397285215 LMissing descriptors for photos 1397285215 and 1397285219 LMissing descriptors for photos 1397285219 and 1397285222 LMissing descriptors for photos 1397285222 and 1397285226 LMissing descriptors for photos 1397285226 and 1397285227 LMissing descriptors for photos 1397285227 and 1397285228 LMissing descriptors for photos 1397285228 and 1397285244 LMissing descriptors for photos 1397285244 and 1397285248 LMissing descriptors for photos 1397285248 and 1397285250 LMissing descriptors for photos 1397285250 and 1397285251 LMissing descriptors for photos 1397285251 and 1397285252 LMissing descriptors for photos 1397285252 and 1397285260 LMissing descriptors for photos 1397285260 and 1397285301 LMissing descriptors for photos 1397285301 and 1397285304 LMissing descriptors for photos 1397285304 and 1397285305 LMissing descriptors for photos 1397285305 and 1397285307 LMissing descriptors for photos 1397285307 and 1397285309 LMissing descriptors for photos 1397285309 and 1397285312 LMissing descriptors for photos 1397285312 and 1397285331 LMissing descriptors for photos 1397285331 and 1397285334 LMissing descriptors for photos 1397285334 and 1397285336 LMissing descriptors for photos 1397285336 and 1397285338 LMissing descriptors for photos 1397285338 and 1397285341 LMissing descriptors for photos 1397285341 and 1397285347 LMissing descriptors for photos 1397285347 and 1397285360 LMissing descriptors for photos 1397285360 and 1397285363 LMissing descriptors for photos 1397285363 and 1397285365 LMissing descriptors for photos 1397285365 and 1397285370 LMissing descriptors for photos 1397285370 and 1397285375 LMissing descriptors for photos 1397285375 and 1397285377 LMissing descriptors for photos 1397285377 and 1397285386 LMissing descriptors for photos 1397285386 and 1397285389 LMissing descriptors for photos 1397285389 and 1397285393 LMissing descriptors for photos 1397285393 and 1397285396 LMissing descriptors for photos 1397285396 and 1397285399 LMissing descriptors for photos 1397285399 and 1397285402 LMissing descriptors for photos 1397285402 and 1397285440 LMissing descriptors for photos 1397285440 and 1397285444 LMissing descriptors for photos 1397285444 and 1397285446 LMissing descriptors for photos 1397285446 and 1397285447 LMissing descriptors for photos 1397285447 and 1397285449 LMissing descriptors for photos 1397285449 and 1397285452 LMissing descriptors for photos 1397285452 and 1397285500 LMissing descriptors for photos 1397285500 and 1397285503 LMissing descriptors for photos 1397285503 and 1397285504 LMissing descriptors for photos 1397285504 and 1397285506 LMissing descriptors for photos 1397285506 and 1397285507 LMissing descriptors for photos 1397285507 and 1397285509 LMissing descriptors for photos 1397285509 and 1397285604 LMissing descriptors for photos 1397285604 and 1397285628 LMissing descriptors for photos 1397285628 and 1397285633 LMissing descriptors for photos 1397285633 and 1397285661 LMissing descriptors for photos 1397285661 and 1397285663 LMissing descriptors for photos 1397285663 and 1397285691 LMissing descriptors for photos 1397285691 and 1397285805 LMissing descriptors for photos 1397285805 and 1397285810 LMissing descriptors for photos 1397285810 and 1397285812 LMissing descriptors for photos 1397285812 and 1397285830 LMissing descriptors for photos 1397285830 and 1397285833 LMissing descriptors for photos 1397285833 and 1397285836 LMissing descriptors for photos 1397285836 and 1397285921 LMissing descriptors for photos 1397285921 and 1397285922 LMissing descriptors for photos 1397285922 and 1397285925 LMissing descriptors for photos 1397285925 and 1397285929 LMissing descriptors for photos 1397285929 and 1397285933 LMissing descriptors for photos 1397285933 and 1397285937 LMissing descriptors for photos 1397285937 and 1397285948 LMissing descriptors for photos 1397285948 and 1397285952 LMissing descriptors for photos 1397285952 and 1397285955 LMissing descriptors for photos 1397285955 and 1397285958 LMissing descriptors for photos 1397285958 and 1397285960 LMissing descriptors for photos 1397285960 and 1397285963 LMissing descriptors for photos 1397285963 and 1397286257 LMissing descriptors for photos 1397286257 and 1397286306 LMissing descriptors for photos 1397286306 and 1397286316 LMissing descriptors for photos 1397286316 and 1397286321 LMissing descriptors for photos 1397286321 and 1397286332 LMissing descriptors for photos 1397286332 and 1397286334 LMissing descriptors for photos 1397286334 and 1397286371 LMissing descriptors for photos 1397286371 and 1397286374 LMissing descriptors for photos 1397286374 and 1397286397 LMissing descriptors for photos 1397286397 and 1397286400 LMissing descriptors for photos 1397286400 and 1397286404 LMissing descriptors for photos 1397286404 and 1397286408 LMissing descriptors for photos 1397286408 and 1397286416 LMissing descriptors for photos 1397286416 and 1397286419 LMissing descriptors for photos 1397286419 and 1397286425 LMissing descriptors for photos 1397286425 and 1397286427 LMissing descriptors for photos 1397286427 and 1397286429 LMissing descriptors for photos 1397286429 and 1397286433 LMissing descriptors for photos 1397286433 and 1397286446 LMissing descriptors for photos 1397286446 and 1397286451 LMissing descriptors for photos 1397286451 and 1397286453 LMissing descriptors for photos 1397286453 and 1397286456 LMissing descriptors for photos 1397286456 and 1397286458 LMissing descriptors for photos 1397286458 and 1397286459 LMissing descriptors for photos 1397286459 and 1397286468 LMissing descriptors for photos 1397286468 and 1397286471 LMissing descriptors for photos 1397286471 and 1397286474 LMissing descriptors for photos 1397286474 and 1397286478 LMissing descriptors for photos 1397286478 and 1397286482 LMissing descriptors for photos 1397286482 and 1397286483 LMissing descriptors for photos 1397286483 and 1397286493 LMissing descriptors for photos 1397286493 and 1397286499 LMissing descriptors for photos 1397286499 and 1397286502 LMissing descriptors for photos 1397286502 and 1397286505 LMissing descriptors for photos 1397286505 and 1397286507 LMissing descriptors for photos 1397286507 and 1397286508 LMissing descriptors for photos 1397286508 and 1397286517 LMissing descriptors for photos 1397286517 and 1397286521 LMissing descriptors for photos 1397286521 and 1397286525 LMissing descriptors for photos 1397286525 and 1397286528 LMissing descriptors for photos 1397286528 and 1397286530 LMissing descriptors for photos 1397286530 and 1397286532 LMissing descriptors for photos 1397286532 and 1397286544 LMissing descriptors for photos 1397286544 and 1397286547 LMissing descriptors for photos 1397286547 and 1397286549 LMissing descriptors for photos 1397286549 and 1397286550 LMissing descriptors for photos 1397286550 and 1397286553 LMissing descriptors for photos 1397286553 and 1397286554 LMissing descriptors for photos 1397286554 and 1397286569 LMissing descriptors for photos 1397286569 and 1397286570 LMissing descriptors for photos 1397286570 and 1397286572 LMissing descriptors for photos 1397286572 and 1397286575 LMissing descriptors for photos 1397286575 and 1397286579 LMissing descriptors for photos 1397286579 and 1397286584 LMissing descriptors for photos 1397286584 and 1397286592 LMissing descriptors for photos 1397286592 and 1397286641 LMissing descriptors for photos 1397286641 and 1397286647 LMissing descriptors for photos 1397286647 and 1397286658 LMissing descriptors for photos 1397286658 and 1397286663 LMissing descriptors for photos 1397286663 and 1397286665 LMissing descriptors for photos 1397286665 and 1397286676 LMissing descriptors for photos 1397286676 and 1397286679 LMissing descriptors for photos 1397286679 and 1397286684 LMissing descriptors for photos 1397286684 and 1397286688 LMissing descriptors for photos 1397286688 and 1397286690 LMissing descriptors for photos 1397286690 and 1397286693 LMissing descriptors for photos 1397286693 and 1397286702 LMissing descriptors for photos 1397286702 and 1397286705 LMissing descriptors for photos 1397286705 and 1397286710 LMissing descriptors for photos 1397286710 and 1397286713 LMissing descriptors for photos 1397286713 and 1397286716 LMissing descriptors for photos 1397286716 and 1397286717 LMissing descriptors for photos 1397286717 and 1397286763 LMissing descriptors for photos 1397286763 and 1397286765 LMissing descriptors for photos 1397286765 and 1397286769 LMissing descriptors for photos 1397286769 and 1397286771 LMissing descriptors for photos 1397286771 and 1397286774 LMissing descriptors for photos 1397286774 and 1397286779 LMissing descriptors for photos 1397286779 and 1397286792 LMissing descriptors for photos 1397286792 and 1397286796 LMissing descriptors for photos 1397286796 and 1397286798 LMissing descriptors for photos 1397286798 and 1397286800 LMissing descriptors for photos 1397286800 and 1397286802 LMissing descriptors for photos 1397286802 and 1397286804 LMissing descriptors for photos 1397286804 and 1397286831 LMissing descriptors for photos 1397286831 and 1397286841 LMissing descriptors for photos 1397286841 and 1397286846 LMissing descriptors for photos 1397286846 and 1397286850 LMissing descriptors for photos 1397286850 and 1397286853 LMissing descriptors for photos 1397286853 and 1397286856 LMissing descriptors for photos 1397286856 and 1397286954 LMissing descriptors for photos 1397286954 and 1397286957 LMissing descriptors for photos 1397286957 and 1397286961 LMissing descriptors for photos 1397286961 and 1397286965 LMissing descriptors for photos 1397286965 and 1397286967 LMissing descriptors for photos 1397286967 and 1397286968 LMissing descriptors for photos 1397286968 and 1397286978 LMissing descriptors for photos 1397286978 and 1397286980 LMissing descriptors for photos 1397286980 and 1397286981 LMissing descriptors for photos 1397286981 and 1397286983 LMissing descriptors for photos 1397286983 and 1397286985 LMissing descriptors for photos 1397286985 and 1397286988 LMissing descriptors for photos 1397286988 and 1397287003 LMissing descriptors for photos 1397287003 and 1397287006 LMissing descriptors for photos 1397287006 and 1397287009 LMissing descriptors for photos 1397287009 and 1397287011 LMissing descriptors for photos 1397287011 and 1397287013 LMissing descriptors for photos 1397287013 and 1397287014 LMissing descriptors for photos 1397287014 and 1397287072 LMissing descriptors for photos 1397287072 and 1397287074 LMissing descriptors for photos 1397287074 and 1397287075 LMissing descriptors for photos 1397287075 and 1397287076 LMissing descriptors for photos 1397287076 and 1397287078 LMissing descriptors for photos 1397287078 and 1397287081 LMissing descriptors for photos 1397287081 and 1397287089 LMissing descriptors for photos 1397287089 and 1397287090 LMissing descriptors for photos 1397287090 and 1397287091 LMissing descriptors for photos 1397287091 and 1397287094 LMissing descriptors for photos 1397287094 and 1397287096 LMissing descriptors for photos 1397287096 and 1397287100 LMissing descriptors for photos 1397287100 and 1397287107 LMissing descriptors for photos 1397287107 and 1397287108 LMissing descriptors for photos 1397287108 and 1397287109 LMissing descriptors for photos 1397287109 and 1397287112 LMissing descriptors for photos 1397287112 and 1397287114 LMissing descriptors for photos 1397287114 and 1397287116 LMissing descriptors for photos 1397287116 and 1397287144 LMissing descriptors for photos 1397287144 and 1397287146 LMissing descriptors for photos 1397287146 and 1397287149 LMissing descriptors for photos 1397287149 and 1397287152 LMissing descriptors for photos 1397287152 and 1397287156 LMissing descriptors for photos 1397287156 and 1397287158 LMissing descriptors for photos 1397287158 and 1397287260 LMissing descriptors for photos 1397287260 and 1397287263 LMissing descriptors for photos 1397287263 and 1397287265 LMissing descriptors for photos 1397287265 and 1397287267 LMissing descriptors for photos 1397287267 and 1397287270 LMissing descriptors for photos 1397287270 and 1397287272 LMissing descriptors for photos 1397287272 and 1397287277 LMissing descriptors for photos 1397287277 and 1397287279 LMissing descriptors for photos 1397287279 and 1397287283 LMissing descriptors for photos 1397287283 and 1397287286 LMissing descriptors for photos 1397287286 and 1397287288 LMissing descriptors for photos 1397287288 and 1397287290 LMissing descriptors for photos 1397287290 and 1397287332 LMissing descriptors for photos 1397287332 and 1397287334 LMissing descriptors for photos 1397287334 and 1397287337 LMissing descriptors for photos 1397287337 and 1397287339 LMissing descriptors for photos 1397287339 and 1397287341 LMissing descriptors for photos 1397287341 and 1397287342 LMissing descriptors for photos 1397287342 and 1397287407 LMissing descriptors for photos 1397287407 and 1397287409 LMissing descriptors for photos 1397287409 and 1397287411 LMissing descriptors for photos 1397287411 and 1397287413 LMissing descriptors for photos 1397287413 and 1397287414 LMissing descriptors for photos 1397287414 and 1397287469 LMissing descriptors for photos 1397287469 and 1397287513 LMissing descriptors for photos 1397287513 and 1397287523 LMissing descriptors for photos 1397287523 and 1397287524 LMissing descriptors for photos 1397287524 and 1397287525 LMissing descriptors for photos 1397287525 and 1397287528 LMissing descriptors for photos 1397287528 and 1397287530 LMissing descriptors for photos 1397287530 and 1397287539 LMissing descriptors for photos 1397287539 and 1397287540 LMissing descriptors for photos 1397287540 and 1397287541 LMissing descriptors for photos 1397287541 and 1397287544 LMissing descriptors for photos 1397287544 and 1397287546 LMissing descriptors for photos 1397287546 and 1397287550 LMissing descriptors for photos 1397287550 and 1397287555 LMissing descriptors for photos 1397287555 and 1397287557 LMissing descriptors for photos 1397287557 and 1397287561 LMissing descriptors for photos 1397287561 and 1397287564 LMissing descriptors for photos 1397287564 and 1397287567 LMissing descriptors for photos 1397287567 and 1397287569 LMissing descriptors for photos 1397287569 and 1397287576 LMissing descriptors for photos 1397287576 and 1397287578 LMissing descriptors for photos 1397287578 and 1397287582 LMissing descriptors for photos 1397287582 and 1397287586 LMissing descriptors for photos 1397287586 and 1397287588 LMissing descriptors for photos 1397287588 and 1397287590 LMissing descriptors for photos 1397287590 and 1397287593 LMissing descriptors for photos 1397287593 and 1397287595 LMissing descriptors for photos 1397287595 and 1397287599 LMissing descriptors for photos 1397287599 and 1397287602 LMissing descriptors for photos 1397287602 and 1397287606 LMissing descriptors for photos 1397287606 and 1397287608 LMissing descriptors for photos 1397287608 and 1397287616 LMissing descriptors for photos 1397287616 and 1397287618 LMissing descriptors for photos 1397287618 and 1397287621 LMissing descriptors for photos 1397287621 and 1397287625 LMissing descriptors for photos 1397287625 and 1397287656 LMissing descriptors for photos 1397287656 and 1397287780 LMissing descriptors for photos 1397287780 and 1397287880 LMissing descriptors for photos 1397287880 and 1397287896 LMissing descriptors for photos 1397287896 and 1397287905 LMissing descriptors for photos 1397287905 and 1397287908 LMissing descriptors for photos 1397287908 and 1397287937 LMissing descriptors for photos 1397287937 and 1397287938 LMissing descriptors for photos 1397287938 and 1397287969 LMissing descriptors for photos 1397287969 and 1397287971 LMissing descriptors for photos 1397287971 and 1397287975 LMissing descriptors for photos 1397287975 and 1397287984 LMissing descriptors for photos 1397287984 and 1397287985 LMissing descriptors for photos 1397287985 and 1397287986 LMissing descriptors for photos 1397287986 and 1397288021 LMissing descriptors for photos 1397288021 and 1397288023 LMissing descriptors for photos 1397288023 and 1397288024 LMissing descriptors for photos 1397288024 and 1397288025 LMissing descriptors for photos 1397288025 and 1397288027 LMissing descriptors for photos 1397288027 and 1397288029 LMissing descriptors for photos 1397288029 and 1397288037 LMissing descriptors for photos 1397288037 and 1397288038 LMissing descriptors for photos 1397288038 and 1397288039 LMissing descriptors for photos 1397288039 and 1397288042 LMissing descriptors for photos 1397288042 and 1397288044 LMissing descriptors for photos 1397288044 and 1397288048 LMissing descriptors for photos 1397288048 and 1397288053 LMissing descriptors for photos 1397288053 and 1397288054 LMissing descriptors for photos 1397288054 and 1397288056 LMissing descriptors for photos 1397288056 and 1397288058 LMissing descriptors for photos 1397288058 and 1397288060 LMissing descriptors for photos 1397288060 and 1397288064 LMissing descriptors for photos 1397288064 and 1397288069 LMissing descriptors for photos 1397288069 and 1397288070 LMissing descriptors for photos 1397288070 and 1397288073 LMissing descriptors for photos 1397288073 and 1397288076 LMissing descriptors for photos 1397288076 and 1397288079 LMissing descriptors for photos 1397288079 and 1397288083 LMissing descriptors for photos 1397288083 and 1397288101 LMissing descriptors for photos 1397288101 and 1397288103 LMissing descriptors for photos 1397288103 and 1397288106 LMissing descriptors for photos 1397288106 and 1397288108 LMissing descriptors for photos 1397288108 and 1397288110 LMissing descriptors for photos 1397288110 and 1397288157 LMissing descriptors for photos 1397288157 and 1397288172 LMissing descriptors for photos 1397288172 and 1397288175 LMissing descriptors for photos 1397288175 and 1397288177 LMissing descriptors for photos 1397288177 and 1397288178 LMissing descriptors for photos 1397288178 and 1397288181 LMissing descriptors for photos 1397288181 and 1397288183 LMissing descriptors for photos 1397288183 and 1397288212 LMissing descriptors for photos 1397288212 and 1397288214 LMissing descriptors for photos 1397288214 and 1397288216 LMissing descriptors for photos 1397288216 and 1397288234 LMissing descriptors for photos 1397288234 and 1397288261 LMissing descriptors for photos 1397288261 and 1397288262 LMissing descriptors for photos 1397288262 and 1397288272 LMissing descriptors for photos 1397288272 and 1397288276 LMissing descriptors for photos 1397288276 and 1397288278 LMissing descriptors for photos 1397288278 and 1397288282 LMissing descriptors for photos 1397288282 and 1397288286 LMissing descriptors for photos 1397288286 and 1397288288 LMissing descriptors for photos 1397288288 and 1397288295 LMissing descriptors for photos 1397288295 and 1397288298 LMissing descriptors for photos 1397288298 and 1397288300 LMissing descriptors for photos 1397288300 and 1397288302 LMissing descriptors for photos 1397288302 and 1397288304 LMissing descriptors for photos 1397288304 and 1397288307 LMissing descriptors for photos 1397288307 and 1397288313 LMissing descriptors for photos 1397288313 and 1397288317 LMissing descriptors for photos 1397288317 and 1397288320 LMissing descriptors for photos 1397288320 and 1397288321 LMissing descriptors for photos 1397288321 and 1397288322 LMissing descriptors for photos 1397288322 and 1397288323 LMissing descriptors for photos 1397288323 and 1397288333 LMissing descriptors for photos 1397288333 and 1397288335 LMissing descriptors for photos 1397288335 and 1397288337 LMissing descriptors for photos 1397288337 and 1397288339 LMissing descriptors for photos 1397288339 and 1397288340 LMissing descriptors for photos 1397288340 and 1397288341 LMissing descriptors for photos 1397288341 and 1397288409 LMissing descriptors for photos 1397288409 and 1397288435 LMissing descriptors for photos 1397288435 and 1397288448 LMissing descriptors for photos 1397288448 and 1397288450 LMissing descriptors for photos 1397288450 and 1397288467 LMissing descriptors for photos 1397288467 and 1397288473 LMissing descriptors for photos 1397288473 and 1397288506 LMissing descriptors for photos 1397288506 and 1397288508 LMissing descriptors for photos 1397288508 and 1397288510 LMissing descriptors for photos 1397288510 and 1397288513 L 21112025 Removing 32 photos because of the 'same image' condition Total on : 3239.460466 list_time_on Total off : 240.00694099999998 list_time_off dist_desc begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 728 time used for this insertion : 0.3277397155761719 photos_removed : len 32 elapsed_time : remove_photo_duplicate 0.3890666961669922 To do, maybe not at the correct place ! ...................................................................L......................................................................................................L.L.L.L.L.L.L.L.L.L.L.L.......L...L.L.L.L.L.L.L.L.L.L........L............................................L.....................L......L.L.L.....................................L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.Lforce hashtag to JRM elapsed_time : CREATE_PORT_BATCH_BY_HOUR 0.025496244430541992 NUMBER BATCH : 4 list_ponderation used : [1e-05, 1e-05, 1e-05, 1e-05, 1e-05] , list_hashtag_class_create_as_list : ['jrm'] result_one_balle_Type_JRM:{'day': '21112025', 'map_nb_amount': {0: 10, 1: 73, 2: 4, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 839.9256939888, 'nb_balles_papier': 0.0008700000000000016, 'begin_time_port': 'image_21112025_10_00_02_843274m0.jpg 1e-05 for time 1, id_amount 2 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 0.0008700000000000016 result_one_balle_Type_JRM:{'day': '21112025', 'map_nb_amount': {0: 15, 1: 62, 2: 6, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 840.7242939472198, 'nb_balles_papier': 0.0008300000000000015, 'begin_time_port': 'image_21112025_10_15_02_489811m0.jpg 1e-05 for time 1, id_amount 3 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 0.0008300000000000015 result_one_balle_Type_JRM:{'day': '21112025', 'map_nb_amount': {0: 15, 1: 64, 2: 2, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 839.2303371429443, 'nb_balles_papier': 0.0008100000000000015, 'begin_time_port': 'image_21112025_10_30_03_029121m0.jpg 1e-05 for time 1, id_amount 2 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 0.0008100000000000015 LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLresult_one_balle_Type_JRM:{'day': '21112025', 'map_nb_amount': {0: 13, 1: 71, 2: 4, 3: 2, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 840.035532951355, 'nb_balles_papier': 0.0009000000000000017, 'begin_time_port': 'image_21112025_10_45_03_061543m0.jpg 1e-05 for time 1, id_amount 1 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 0.0009000000000000017 We have rejected 0 photos because of the batch_size condition ! NUMBER BATCH list_of_portfolios_to_create : 4 list_same_port_ids : [28776702] find same portfolio which already exist 28776702 , we will use it list_same_port_ids : [28778850] find same portfolio which already exist 28778850 , we will use it list_same_port_ids : [28812169] find same portfolio which already exist 28812169 , we will use it list_same_port_ids : [] Qualite : 0.13192845900389 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=28776702 AND mptpi.`type`=3726 To do Qualite : 0.10716566765619162 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=28778850 AND mptpi.`type`=3726 To do Qualite : 0.13009949224421644 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=28812169 AND mptpi.`type`=3726 To do # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=29080290 AND mptpi.`type`=3726 To do elapsed_time : count_nb_balles_and_create_portfolio 3.9188268184661865 # DISPLAY ALL COLLECTED DATA : {'21112025': {'nb_upload': 728, 'nb_taggue_class': 728, 'nb_taggue_densite': 728, 'nb_descriptors': 373}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1397288513, 1397288510, 1397288508, 1397288506, 1397288473, 1397288467, 1397288450, 1397288448, 1397288435, 1397288409, 1397288341, 1397288340, 1397288339, 1397288337, 1397288335, 1397288333, 1397288323, 1397288322, 1397288321, 1397288320, 1397288317, 1397288313, 1397288307, 1397288304, 1397288302, 1397288300, 1397288298, 1397288295, 1397288288, 1397288286, 1397288282, 1397288278, 1397288276, 1397288272, 1397288262, 1397288261, 1397288234, 1397288216, 1397288214, 1397288212, 1397288183, 1397288181, 1397288178, 1397288177, 1397288175, 1397288172, 1397288157, 1397288110, 1397288108, 1397288106, 1397288103, 1397288101, 1397288083, 1397288079, 1397288076, 1397288073, 1397288070, 1397288069, 1397288064, 1397288060, 1397288058, 1397288056, 1397288054, 1397288053, 1397288048, 1397288044, 1397288042, 1397288039, 1397288038, 1397288037, 1397288029, 1397288027, 1397288025, 1397288024, 1397288023, 1397288021, 1397287986, 1397287985, 1397287984, 1397287975, 1397287971, 1397287969, 1397287938, 1397287937, 1397287908, 1397287905, 1397287896, 1397287880, 1397287780, 1397287656, 1397287625, 1397287621, 1397287618, 1397287616, 1397287608, 1397287606, 1397287602, 1397287599, 1397287595, 1397287593] Looping around the photos to save general results len do output : 1 /28772302Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288513', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288510', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288508', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288506', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288473', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288467', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288450', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288448', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288435', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288409', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288341', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288340', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288339', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288337', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288335', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288333', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288323', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288322', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288321', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288320', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288317', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288313', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288307', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288304', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288302', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288300', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288298', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288295', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288288', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288286', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288282', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288278', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288276', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288272', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288262', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288261', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288234', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288216', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288214', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288212', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288183', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288181', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288178', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288177', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288175', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288172', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288157', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288110', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288108', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288106', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288103', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288101', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288083', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288079', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288076', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288073', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288070', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288069', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288064', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288060', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288058', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288056', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288054', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288053', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288048', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288044', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288042', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288039', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288038', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288037', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288029', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288027', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288025', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288024', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288023', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397288021', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397287986', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397287985', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397287984', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397287975', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397287971', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397287969', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397287938', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397287937', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397287908', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397287905', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397287896', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397287880', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397287780', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397287656', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397287625', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397287621', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397287618', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397287616', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397287608', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397287606', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397287602', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397287599', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397287595', None, None, None, None, None, '4169169') ('4189', None, None, None, None, None, None, None, '4169169') ('4189', '28772302', '1397287593', None, None, None, None, None, '4169169') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 101 time used for this insertion : 0.04355335235595703 save_final save missing photos in datou_result : time spend for datou_step_exec : 36.1261727809906 time spend to save output : 0.04439091682434082 total time spend for step 1 : 36.17056369781494 caffe_path_current : About to save ! 2 After save, about to update current ! ret : 2 len(input) + len(total_photo_id_missing) : 1 set_done_treatment 2.35user 0.81system 0:38.62elapsed 8%CPU (0avgtext+0avgdata 112544maxresident)k 2512inputs+3144outputs (4major+53787minor)pagefaults 0swaps