python /home/admin/mtr/script_for_cron.py -j default -m 20 -a 'python3 ~/workarea/git/Velours/python/prod/datou.py -j batch_current -C 2944292' -s traitement_sts -M 0 -S 0 -U 100,80,95 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/home/admin/workarea/git/apy', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 773298 load datou : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : step 0 init_dummy_multi_datou is not linked in the step_by_step architecture ! WARNING : step 1294 init_dummy_multi_datou is not linked in the step_by_step architecture ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : (photo_id, hashtag_id, score_max) was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 list_input_json: {} Current got : datou_id : 3142, datou_cur_ids : ['2944292'] with mtr_portfolio_ids : ['23234922'] and first list_photo_ids : [] new path : /proc/773298/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : split_time_score over limit max, limiting to limit_max 100 list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB time to download the photos : 0.09490680694580078 About to test input to load Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 1 step1:split_time_score Fri May 23 10:36:04 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec begin split time score 2022-04-13 10:29:59 0 TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 3379, 'mtr_user_id': 31, 'name': 'learn_classif_flux_maj_generique_effnet_v2_s_02062022', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'aluminium,ela,film_pedb,flux_dev,jrm,pcm,pcnc,pehd_pp,pet_clair,refus,tapis_vide', 'svm_portfolios_learning': '5515864,5515840,5515844,5515850,6244400,6237996,6237998,5515847,5515841,5515868,5515866', 'photo_hashtag_type': 4374, 'photo_desc_type': 5680, 'type_classification': 'tf_classification2', 'hashtag_id_list': '493546845,492741797,2107760237,2107760238,495916461,560181804,1284539308,2107760239,2107755846,538914404,2107748999'}] thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('00', 19), ('01', 11), ('02', 27), ('03', 18), ('04', 9), ('05', 4), ('06', 7), ('07', 8), ('08', 9), ('09', 5), ('10', 9), ('11', 10), ('12', 10), ('13', 8), ('14', 44), ('17', 5), ('16', 15), ('15', 5), ('18', 3), ('19', 2), ('20', 4), ('22', 2), ('23', 19)) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {1: 148} 22052025 23234922 Nombre de photos uploadées : 253 / 23040 (1%) 22052025 23234922 Nombre de photos taguées (types de déchets): 148 / 253 (58%) 22052025 23234922 Nombre de photos taguées (volume) : 148 / 253 (58%) elapsed_time : load_data_split_time_score 1.9073486328125e-06 elapsed_time : order_list_meta_photo_and_scores 6.818771362304688e-05 ????????????????????????????????????????????????????????????????????????????????????????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.012516498565673828 elapsed_time : insert_dashboard_record_day_entry 0.025187015533447266 Creating list_photo_by_hashtags Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None elapsed_time : list_photo_by_hashtags 0.035384416580200195 Creating list_photo_total elapsed_time : select_descriptors 10.17724084854126 22052025 23234922 Nombre de photos avec descriptors (type 5680) : 147 / 252 (58%) ERROR : Hum hum, what can we do for different size of descriptors (ignore the difference ) : 0 vs 1280 photo_id : 1360247149 photo_id_prec : 0 0:00:00|ON:ERROR : Hum hum, what can we do for different size of descriptors (ignore the difference ) : 1280 vs 0 photo_id : 1360565618 photo_id_prec : 1360565376 Missing descriptors for photos 1360565618 and 1360568997 Missing descriptors for photos 1360568997 and 1360569754 Missing descriptors for photos 1360569754 and 1360570072 Missing descriptors for photos 1360570072 and 1360571837 Missing descriptors for photos 1360571837 and 1360572202 Missing descriptors for photos 1360572202 and 1360574569 Missing descriptors for photos 1360574569 and 1360593628 Missing descriptors for photos 1360593628 and 1360593574 Missing descriptors for photos 1360593574 and 1360593564 Missing descriptors for photos 1360593564 and 1360593559 Missing descriptors for photos 1360593559 and 1360593555 Missing descriptors for photos 1360593555 and 1360593552 Missing descriptors for photos 1360593552 and 1360591908 Missing descriptors for photos 1360591908 and 1360591858 Missing descriptors for photos 1360591858 and 1360591812 Missing descriptors for photos 1360591812 and 1360591772 Missing descriptors for photos 1360591772 and 1360591732 Missing descriptors for photos 1360591732 and 1360591694 Missing descriptors for photos 1360591694 and 1360589255 Missing descriptors for photos 1360589255 and 1360589200 Missing descriptors for photos 1360589200 and 1360589158 Missing descriptors for photos 1360589158 and 1360589111 Missing descriptors for photos 1360589111 and 1360589072 Missing descriptors for photos 1360589072 and 1360589041 Missing descriptors for photos 1360589041 and 1360587330 Missing descriptors for photos 1360587330 and 1360587292 Missing descriptors for photos 1360587292 and 1360587253 Missing descriptors for photos 1360587253 and 1360587198 Missing descriptors for photos 1360587198 and 1360587141 Missing descriptors for photos 1360587141 and 1360587101 Missing descriptors for photos 1360587101 and 1360585163 Missing descriptors for photos 1360585163 and 1360585156 Missing descriptors for photos 1360585156 and 1360585152 Missing descriptors for photos 1360585152 and 1360585145 Missing descriptors for photos 1360585145 and 1360585140 Missing descriptors for photos 1360585140 and 1360585137 Missing descriptors for photos 1360585137 and 1360583146 Missing descriptors for photos 1360583146 and 1360583143 Missing descriptors for photos 1360583143 and 1360583139 Missing descriptors for photos 1360583139 and 1360583134 Missing descriptors for photos 1360583134 and 1360583128 Missing descriptors for photos 1360583128 and 1360583116 Missing descriptors for photos 1360583116 and 1360581002 Missing descriptors for photos 1360581002 and 1360580970 Missing descriptors for photos 1360580970 and 1360580941 Missing descriptors for photos 1360580941 and 1360580913 Missing descriptors for photos 1360580913 and 1360579999 Missing descriptors for photos 1360579999 and 1360580882 Missing descriptors for photos 1360580882 and 1360579774 Missing descriptors for photos 1360579774 and 1360579768 Missing descriptors for photos 1360579768 and 1360579745 Missing descriptors for photos 1360579745 and 1360579727 Missing descriptors for photos 1360579727 and 1360579713 Missing descriptors for photos 1360579713 and 1360579677 Missing descriptors for photos 1360579677 and 1360579145 Missing descriptors for photos 1360579145 and 1360579112 Missing descriptors for photos 1360579112 and 1360579084 Missing descriptors for photos 1360579084 and 1360579047 Missing descriptors for photos 1360579047 and 1360579019 Missing descriptors for photos 1360579019 and 1360579011 Missing descriptors for photos 1360579011 and 1360578474 Missing descriptors for photos 1360578474 and 1360578468 Missing descriptors for photos 1360578468 and 1360578462 Missing descriptors for photos 1360578462 and 1360578455 Missing descriptors for photos 1360578455 and 1360578450 Missing descriptors for photos 1360578450 and 1360578447 Missing descriptors for photos 1360578447 and 1360577279 Missing descriptors for photos 1360577279 and 1360577244 Missing descriptors for photos 1360577244 and 1360577233 Missing descriptors for photos 1360577233 and 1360577228 Missing descriptors for photos 1360577228 and 1360577224 Missing descriptors for photos 1360577224 and 1360577219 Missing descriptors for photos 1360577219 and 1360579998 Missing descriptors for photos 1360579998 and 1360580855 Missing descriptors for photos 1360580855 and 1360593548 Missing descriptors for photos 1360593548 and 1360594525 Missing descriptors for photos 1360594525 and 1360595420 Missing descriptors for photos 1360595420 and 1360603801 Missing descriptors for photos 1360603801 and 1360605718 Missing descriptors for photos 1360605718 and 1360622183 Missing descriptors for photos 1360622183 and 1360625315 Missing descriptors for photos 1360625315 and 1360627637 Missing descriptors for photos 1360627637 and 1360637838 Missing descriptors for photos 1360637838 and 1360668647 Missing descriptors for photos 1360668647 and 1360675091 Missing descriptors for photos 1360675091 and 1360678109 Missing descriptors for photos 1360678109 and 1360678660 Missing descriptors for photos 1360678660 and 1360678651 Missing descriptors for photos 1360678651 and 1360678744 Missing descriptors for photos 1360678744 and 1360678804 Missing descriptors for photos 1360678804 and 1360679084 Missing descriptors for photos 1360679084 and 1360679325 Missing descriptors for photos 1360679325 and 1360679838 Missing descriptors for photos 1360679838 and 1360680322 Missing descriptors for photos 1360680322 and 1360680439 Missing descriptors for photos 1360680439 and 1360680482 Missing descriptors for photos 1360680482 and 1360680564 Missing descriptors for photos 1360680564 and 1360680643 Missing descriptors for photos 1360680643 and 1360680692 Missing descriptors for photos 1360680692 and 1360680768 Missing descriptors for photos 1360680768 and 1360680934 Missing descriptors for photos 1360680934 and 1360680928 Missing descriptors for photos 1360680928 and 1360680923 Missing descriptors for photos 1360680923 and 1360680983 22052025 Removing 0 photos because of the 'same image' condition Total on : 0 Total off : 0.0 list_time_off Warning in study_and_display_distrib_list : min=max : 0.0 0.0 dist_desc begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 253 time used for this insertion : 0.09812140464782715 photos_removed : len 0 elapsed_time : remove_photo_duplicate 0.13658452033996582 To do, maybe not at the correct place ! .............................................................................................................................................................................................................................................................force hashtag to refus elapsed_time : CREATE_PORT_BATCH_BY_HOUR 0.013084650039672852 NUMBER BATCH : 23 list_ponderation used : [1e-05, 1e-05, 1e-05, 1e-05, 1e-05] , list_hashtag_class_create_as_list : ['refus'] result_one_balle_Type_refus:{'day': '22052025', 'map_nb_amount': {0: 1, 1: 13, 2: 5, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 3360.0001060962677, 'nb_balles_papier': 0.00019, 'begin_time_port': 'image_22052025_00_03_35_029909m0.jpg 1e-05 for time 1, id_amount 2 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 0.00019 result_one_balle_Type_refus:{'day': '22052025', 'map_nb_amount': {0: 0, 1: 11, 2: 0, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 2010.9999952316284, 'nb_balles_papier': 0.00011, 'begin_time_port': 'image_22052025_01_25_34_030007m0.jpg 1e-05 for time 1, id_amount 2 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 0.00011 result_one_balle_Type_refus:{'day': '22052025', 'map_nb_amount': {0: 1, 1: 25, 2: 1, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 3539.9989819526672, 'nb_balles_papier': 0.00027000000000000006, 'begin_time_port': 'image_22052025_02_00_35_030927m0.jpg 1e-05 for time 1, id_amount 2 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 0.00027000000000000006 result_one_balle_Type_refus:{'day': '22052025', 'map_nb_amount': {0: 4, 1: 14, 2: 0, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 2399.9989948272705, 'nb_balles_papier': 0.00018, 'begin_time_port': 'image_22052025_03_06_05_030950m0.jpg 1e-05 for time 1, id_amount 2 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 0.00018 result_one_balle_Type_refus:{'day': '22052025', 'map_nb_amount': {0: 1, 1: 8, 2: 0, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 1049.9999940395355, 'nb_balles_papier': 9e-05, 'begin_time_port': 'image_22052025_04_05_35_029935m0.jpg 1e-05 for time 1, id_amount 2 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 9e-05 result_one_balle_Type_refus:{'day': '22052025', 'map_nb_amount': {0: 0, 1: 4, 2: 0, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 2519.9999787807465, 'nb_balles_papier': 4e-05, 'begin_time_port': 'image_22052025_05_17_35_029966m0.jpg 1e-05 for time 1, id_amount 2 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 4e-05 result_one_balle_Type_refus:{'day': '22052025', 'map_nb_amount': {0: 0, 1: 7, 2: 0, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 2760.0000519752502, 'nb_balles_papier': 7.000000000000001e-05, 'begin_time_port': 'image_22052025_06_09_05_029931m0.jpg 1e-05 for time 1, id_amount 2 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 7.000000000000001e-05 result_one_balle_Type_refus:{'day': '22052025', 'map_nb_amount': {0: 0, 1: 6, 2: 0, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 1979.999925851822, 'nb_balles_papier': 7.000000000000001e-05, 'begin_time_port': 'image_22052025_07_11_05_030002m0.jpg 1e-05 for time 1, id_amount 2 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 7.000000000000001e-05 result_one_balle_Type_refus:{'day': '22052025', 'map_nb_amount': {0: 1, 1: 6, 2: 2, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 2670.0000381469727, 'nb_balles_papier': 9e-05, 'begin_time_port': 'image_22052025_08_04_35_029908m0.jpg 1e-05 for time 1, id_amount 3 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 9e-05 result_one_balle_Type_refus:{'day': '22052025', 'map_nb_amount': {0: 0, 1: 3, 2: 2, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 2279.999830007553, 'nb_balles_papier': 5e-05, 'begin_time_port': 'image_22052025_09_19_35_030122m0.jpg 1e-05 for time 1, id_amount 2 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 5e-05 result_one_balle_Type_refus:{'day': '22052025', 'map_nb_amount': {0: 2, 1: 7, 2: 0, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 3510.000013113022, 'nb_balles_papier': 9e-05, 'begin_time_port': 'image_22052025_10_01_05_029923m0.jpg 1e-05 for time 1, id_amount 2 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 9e-05 result_one_balle_Type_refus:{'day': '22052025', 'map_nb_amount': {0: 0, 1: 9, 2: 1, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 1769.9999759197235, 'nb_balles_papier': 0.0001, 'begin_time_port': 'image_22052025_11_07_05_029999m0.jpg 1e-05 for time 1, id_amount 2 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 0.0001 result_one_balle_Type_refus:{'day': '22052025', 'map_nb_amount': {0: 0, 1: 8, 2: 2, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 2849.9999799728394, 'nb_balles_papier': 0.0001, 'begin_time_port': 'image_22052025_12_00_35_030011m0.jpg 1e-05 for time 1, id_amount 3 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 0.0001 ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info result_one_balle_Type_refus:{'day': '22052025', 'map_nb_amount': {0: 0, 1: 2, 2: 0, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 2130.0000109672546, 'nb_balles_papier': 2e-05, 'begin_time_port': 'image_22052025_13_10_35_029969m0.jpg 1e-05 for time 1, id_amount 2 this amount prod time diff : 1e-05'} Production hashtag (incorrect ponderation at 20-10-18) : 2e-05 We filter photos on hashtag condition ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! We filter photos on hashtag condition ! We have rejected 0 photos because of the batch_size condition ! NUMBER BATCH list_of_portfolios_to_create : 14 list_same_port_ids : [23260219] find same portfolio which already exist 23260219 , we will use it list_same_port_ids : [23260222] find same portfolio which already exist 23260222 , we will use it list_same_port_ids : [23260224] find same portfolio which already exist 23260224 , we will use it list_same_port_ids : [23260226] find same portfolio which already exist 23260226 , we will use it list_same_port_ids : [23260229] find same portfolio which already exist 23260229 , we will use it list_same_port_ids : [23260231] find same portfolio which already exist 23260231 , we will use it list_same_port_ids : [23260234] find same portfolio which already exist 23260234 , we will use it list_same_port_ids : [23260235] find same portfolio which already exist 23260235 , we will use it list_same_port_ids : [23260237] find same portfolio which already exist 23260237 , we will use it list_same_port_ids : [23263457] find same portfolio which already exist 23263457 , we will use it list_same_port_ids : [23263459] find same portfolio which already exist 23263459 , we will use it list_same_port_ids : [23265368] find same portfolio which already exist 23265368 , we will use it list_same_port_ids : [23267379] find same portfolio which already exist 23267379 , we will use it list_same_port_ids : [23270896] find same portfolio which already exist 23270896 , we will use it # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 10383 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 10391 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10392 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 10387 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 10387 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 10390 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 10390 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11838 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11838 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 10386 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 10385 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 10385 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 10393 send_mail_cod have less outputs used (0) than in the step definition (1) : some outputs may be not used ! Step 10430 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 3 of step 10385 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 10383 doesn't seem to be define in the database( WARNING : type of input 2 of step 10387 doesn't seem to be define in the database( WARNING : output 1 of step 10383 have datatype=2 whereas input 1 of step 10390 have datatype=7 WARNING : type of output 2 of step 10390 doesn't seem to be define in the database( WARNING : type of input 1 of step 10384 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 10386 have datatype=10 whereas input 3 of step 10393 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10391 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of output 1 of step 10392 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of input 5 of step 10393 doesn't seem to be define in the database( WARNING : output 0 of step 11836 have datatype=11 whereas input 5 of step 10393 have datatype=None WARNING : output 0 of step 10386 have datatype=10 whereas input 0 of step 11836 have datatype=18 WARNING : type of output 3 of step 11838 doesn't seem to be define in the database( WARNING : type of input 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 2 of step 11838 doesn't seem to be define in the database( WARNING : output 1 of step 10384 have datatype=7 whereas input 2 of step 11838 have datatype=None WARNING : output 0 of step 10390 have datatype=1 whereas input 0 of step 10384 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=23260219 AND mptpi.`type`=4213 To do # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 10383 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 10391 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10392 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 10387 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 10387 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 10390 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 10390 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11838 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11838 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 10386 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 10385 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 10385 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 10393 send_mail_cod have less outputs used (0) than in the step definition (1) : some outputs may be not used ! Step 10430 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 3 of step 10385 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 10383 doesn't seem to be define in the database( WARNING : type of input 2 of step 10387 doesn't seem to be define in the database( WARNING : output 1 of step 10383 have datatype=2 whereas input 1 of step 10390 have datatype=7 WARNING : type of output 2 of step 10390 doesn't seem to be define in the database( WARNING : type of input 1 of step 10384 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 10386 have datatype=10 whereas input 3 of step 10393 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10391 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of output 1 of step 10392 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of input 5 of step 10393 doesn't seem to be define in the database( WARNING : output 0 of step 11836 have datatype=11 whereas input 5 of step 10393 have datatype=None WARNING : output 0 of step 10386 have datatype=10 whereas input 0 of step 11836 have datatype=18 WARNING : type of output 3 of step 11838 doesn't seem to be define in the database( WARNING : type of input 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 2 of step 11838 doesn't seem to be define in the database( WARNING : output 1 of step 10384 have datatype=7 whereas input 2 of step 11838 have datatype=None WARNING : output 0 of step 10390 have datatype=1 whereas input 0 of step 10384 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=23260222 AND mptpi.`type`=4213 To do # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 10383 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 10391 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10392 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 10387 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 10387 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 10390 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 10390 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11838 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11838 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 10386 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 10385 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 10385 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 10393 send_mail_cod have less outputs used (0) than in the step definition (1) : some outputs may be not used ! Step 10430 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 3 of step 10385 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 10383 doesn't seem to be define in the database( WARNING : type of input 2 of step 10387 doesn't seem to be define in the database( WARNING : output 1 of step 10383 have datatype=2 whereas input 1 of step 10390 have datatype=7 WARNING : type of output 2 of step 10390 doesn't seem to be define in the database( WARNING : type of input 1 of step 10384 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 10386 have datatype=10 whereas input 3 of step 10393 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10391 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of output 1 of step 10392 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of input 5 of step 10393 doesn't seem to be define in the database( WARNING : output 0 of step 11836 have datatype=11 whereas input 5 of step 10393 have datatype=None WARNING : output 0 of step 10386 have datatype=10 whereas input 0 of step 11836 have datatype=18 WARNING : type of output 3 of step 11838 doesn't seem to be define in the database( WARNING : type of input 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 2 of step 11838 doesn't seem to be define in the database( WARNING : output 1 of step 10384 have datatype=7 whereas input 2 of step 11838 have datatype=None WARNING : output 0 of step 10390 have datatype=1 whereas input 0 of step 10384 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=23260224 AND mptpi.`type`=4213 To do # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 10383 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 10391 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10392 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 10387 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 10387 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 10390 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 10390 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11838 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11838 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 10386 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 10385 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 10385 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 10393 send_mail_cod have less outputs used (0) than in the step definition (1) : some outputs may be not used ! Step 10430 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 3 of step 10385 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 10383 doesn't seem to be define in the database( WARNING : type of input 2 of step 10387 doesn't seem to be define in the database( WARNING : output 1 of step 10383 have datatype=2 whereas input 1 of step 10390 have datatype=7 WARNING : type of output 2 of step 10390 doesn't seem to be define in the database( WARNING : type of input 1 of step 10384 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 10386 have datatype=10 whereas input 3 of step 10393 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10391 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of output 1 of step 10392 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of input 5 of step 10393 doesn't seem to be define in the database( WARNING : output 0 of step 11836 have datatype=11 whereas input 5 of step 10393 have datatype=None WARNING : output 0 of step 10386 have datatype=10 whereas input 0 of step 11836 have datatype=18 WARNING : type of output 3 of step 11838 doesn't seem to be define in the database( WARNING : type of input 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 2 of step 11838 doesn't seem to be define in the database( WARNING : output 1 of step 10384 have datatype=7 whereas input 2 of step 11838 have datatype=None WARNING : output 0 of step 10390 have datatype=1 whereas input 0 of step 10384 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=23260226 AND mptpi.`type`=4213 To do # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 10383 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 10391 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10392 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 10387 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 10387 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 10390 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 10390 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11838 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11838 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 10386 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 10385 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 10385 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 10393 send_mail_cod have less outputs used (0) than in the step definition (1) : some outputs may be not used ! Step 10430 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 3 of step 10385 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 10383 doesn't seem to be define in the database( WARNING : type of input 2 of step 10387 doesn't seem to be define in the database( WARNING : output 1 of step 10383 have datatype=2 whereas input 1 of step 10390 have datatype=7 WARNING : type of output 2 of step 10390 doesn't seem to be define in the database( WARNING : type of input 1 of step 10384 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 10386 have datatype=10 whereas input 3 of step 10393 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10391 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of output 1 of step 10392 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of input 5 of step 10393 doesn't seem to be define in the database( WARNING : output 0 of step 11836 have datatype=11 whereas input 5 of step 10393 have datatype=None WARNING : output 0 of step 10386 have datatype=10 whereas input 0 of step 11836 have datatype=18 WARNING : type of output 3 of step 11838 doesn't seem to be define in the database( WARNING : type of input 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 2 of step 11838 doesn't seem to be define in the database( WARNING : output 1 of step 10384 have datatype=7 whereas input 2 of step 11838 have datatype=None WARNING : output 0 of step 10390 have datatype=1 whereas input 0 of step 10384 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=23260229 AND mptpi.`type`=4213 To do # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 10383 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 10391 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10392 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 10387 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 10387 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 10390 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 10390 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11838 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11838 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 10386 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 10385 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 10385 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 10393 send_mail_cod have less outputs used (0) than in the step definition (1) : some outputs may be not used ! Step 10430 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 3 of step 10385 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 10383 doesn't seem to be define in the database( WARNING : type of input 2 of step 10387 doesn't seem to be define in the database( WARNING : output 1 of step 10383 have datatype=2 whereas input 1 of step 10390 have datatype=7 WARNING : type of output 2 of step 10390 doesn't seem to be define in the database( WARNING : type of input 1 of step 10384 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 10386 have datatype=10 whereas input 3 of step 10393 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10391 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of output 1 of step 10392 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of input 5 of step 10393 doesn't seem to be define in the database( WARNING : output 0 of step 11836 have datatype=11 whereas input 5 of step 10393 have datatype=None WARNING : output 0 of step 10386 have datatype=10 whereas input 0 of step 11836 have datatype=18 WARNING : type of output 3 of step 11838 doesn't seem to be define in the database( WARNING : type of input 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 2 of step 11838 doesn't seem to be define in the database( WARNING : output 1 of step 10384 have datatype=7 whereas input 2 of step 11838 have datatype=None WARNING : output 0 of step 10390 have datatype=1 whereas input 0 of step 10384 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=23260231 AND mptpi.`type`=4213 To do # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 10383 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 10391 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10392 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 10387 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 10387 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 10390 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 10390 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11838 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11838 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 10386 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 10385 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 10385 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 10393 send_mail_cod have less outputs used (0) than in the step definition (1) : some outputs may be not used ! Step 10430 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 3 of step 10385 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 10383 doesn't seem to be define in the database( WARNING : type of input 2 of step 10387 doesn't seem to be define in the database( WARNING : output 1 of step 10383 have datatype=2 whereas input 1 of step 10390 have datatype=7 WARNING : type of output 2 of step 10390 doesn't seem to be define in the database( WARNING : type of input 1 of step 10384 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 10386 have datatype=10 whereas input 3 of step 10393 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10391 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of output 1 of step 10392 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of input 5 of step 10393 doesn't seem to be define in the database( WARNING : output 0 of step 11836 have datatype=11 whereas input 5 of step 10393 have datatype=None WARNING : output 0 of step 10386 have datatype=10 whereas input 0 of step 11836 have datatype=18 WARNING : type of output 3 of step 11838 doesn't seem to be define in the database( WARNING : type of input 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 2 of step 11838 doesn't seem to be define in the database( WARNING : output 1 of step 10384 have datatype=7 whereas input 2 of step 11838 have datatype=None WARNING : output 0 of step 10390 have datatype=1 whereas input 0 of step 10384 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=23260234 AND mptpi.`type`=4213 To do # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 10383 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 10391 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10392 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 10387 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 10387 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 10390 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 10390 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11838 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11838 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 10386 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 10385 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 10385 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 10393 send_mail_cod have less outputs used (0) than in the step definition (1) : some outputs may be not used ! Step 10430 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 3 of step 10385 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 10383 doesn't seem to be define in the database( WARNING : type of input 2 of step 10387 doesn't seem to be define in the database( WARNING : output 1 of step 10383 have datatype=2 whereas input 1 of step 10390 have datatype=7 WARNING : type of output 2 of step 10390 doesn't seem to be define in the database( WARNING : type of input 1 of step 10384 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 10386 have datatype=10 whereas input 3 of step 10393 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10391 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of output 1 of step 10392 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of input 5 of step 10393 doesn't seem to be define in the database( WARNING : output 0 of step 11836 have datatype=11 whereas input 5 of step 10393 have datatype=None WARNING : output 0 of step 10386 have datatype=10 whereas input 0 of step 11836 have datatype=18 WARNING : type of output 3 of step 11838 doesn't seem to be define in the database( WARNING : type of input 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 2 of step 11838 doesn't seem to be define in the database( WARNING : output 1 of step 10384 have datatype=7 whereas input 2 of step 11838 have datatype=None WARNING : output 0 of step 10390 have datatype=1 whereas input 0 of step 10384 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=23260235 AND mptpi.`type`=4213 To do # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 10383 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 10391 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10392 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 10387 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 10387 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 10390 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 10390 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11838 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11838 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 10386 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 10385 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 10385 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 10393 send_mail_cod have less outputs used (0) than in the step definition (1) : some outputs may be not used ! Step 10430 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 3 of step 10385 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 10383 doesn't seem to be define in the database( WARNING : type of input 2 of step 10387 doesn't seem to be define in the database( WARNING : output 1 of step 10383 have datatype=2 whereas input 1 of step 10390 have datatype=7 WARNING : type of output 2 of step 10390 doesn't seem to be define in the database( WARNING : type of input 1 of step 10384 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 10386 have datatype=10 whereas input 3 of step 10393 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10391 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of output 1 of step 10392 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of input 5 of step 10393 doesn't seem to be define in the database( WARNING : output 0 of step 11836 have datatype=11 whereas input 5 of step 10393 have datatype=None WARNING : output 0 of step 10386 have datatype=10 whereas input 0 of step 11836 have datatype=18 WARNING : type of output 3 of step 11838 doesn't seem to be define in the database( WARNING : type of input 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 2 of step 11838 doesn't seem to be define in the database( WARNING : output 1 of step 10384 have datatype=7 whereas input 2 of step 11838 have datatype=None WARNING : output 0 of step 10390 have datatype=1 whereas input 0 of step 10384 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=23260237 AND mptpi.`type`=4213 To do # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 10383 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 10391 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10392 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 10387 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 10387 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 10390 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 10390 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11838 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11838 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 10386 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 10385 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 10385 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 10393 send_mail_cod have less outputs used (0) than in the step definition (1) : some outputs may be not used ! Step 10430 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 3 of step 10385 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 10383 doesn't seem to be define in the database( WARNING : type of input 2 of step 10387 doesn't seem to be define in the database( WARNING : output 1 of step 10383 have datatype=2 whereas input 1 of step 10390 have datatype=7 WARNING : type of output 2 of step 10390 doesn't seem to be define in the database( WARNING : type of input 1 of step 10384 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 10386 have datatype=10 whereas input 3 of step 10393 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10391 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of output 1 of step 10392 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of input 5 of step 10393 doesn't seem to be define in the database( WARNING : output 0 of step 11836 have datatype=11 whereas input 5 of step 10393 have datatype=None WARNING : output 0 of step 10386 have datatype=10 whereas input 0 of step 11836 have datatype=18 WARNING : type of output 3 of step 11838 doesn't seem to be define in the database( WARNING : type of input 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 2 of step 11838 doesn't seem to be define in the database( WARNING : output 1 of step 10384 have datatype=7 whereas input 2 of step 11838 have datatype=None WARNING : output 0 of step 10390 have datatype=1 whereas input 0 of step 10384 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=23263457 AND mptpi.`type`=4213 To do # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 10383 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 10391 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10392 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 10387 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 10387 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 10390 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 10390 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11838 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11838 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 10386 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 10385 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 10385 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 10393 send_mail_cod have less outputs used (0) than in the step definition (1) : some outputs may be not used ! Step 10430 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 3 of step 10385 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 10383 doesn't seem to be define in the database( WARNING : type of input 2 of step 10387 doesn't seem to be define in the database( WARNING : output 1 of step 10383 have datatype=2 whereas input 1 of step 10390 have datatype=7 WARNING : type of output 2 of step 10390 doesn't seem to be define in the database( WARNING : type of input 1 of step 10384 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 10386 have datatype=10 whereas input 3 of step 10393 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10391 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of output 1 of step 10392 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of input 5 of step 10393 doesn't seem to be define in the database( WARNING : output 0 of step 11836 have datatype=11 whereas input 5 of step 10393 have datatype=None WARNING : output 0 of step 10386 have datatype=10 whereas input 0 of step 11836 have datatype=18 WARNING : type of output 3 of step 11838 doesn't seem to be define in the database( WARNING : type of input 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 2 of step 11838 doesn't seem to be define in the database( WARNING : output 1 of step 10384 have datatype=7 whereas input 2 of step 11838 have datatype=None WARNING : output 0 of step 10390 have datatype=1 whereas input 0 of step 10384 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=23263459 AND mptpi.`type`=4213 To do Qualite : 0.09937511841455884 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 10383 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 10391 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10392 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 10387 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 10387 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 10390 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 10390 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11838 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11838 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 10386 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 10385 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 10385 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 10393 send_mail_cod have less outputs used (0) than in the step definition (1) : some outputs may be not used ! Step 10430 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 3 of step 10385 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 10383 doesn't seem to be define in the database( WARNING : type of input 2 of step 10387 doesn't seem to be define in the database( WARNING : output 1 of step 10383 have datatype=2 whereas input 1 of step 10390 have datatype=7 WARNING : type of output 2 of step 10390 doesn't seem to be define in the database( WARNING : type of input 1 of step 10384 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 10386 have datatype=10 whereas input 3 of step 10393 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10391 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of output 1 of step 10392 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of input 5 of step 10393 doesn't seem to be define in the database( WARNING : output 0 of step 11836 have datatype=11 whereas input 5 of step 10393 have datatype=None WARNING : output 0 of step 10386 have datatype=10 whereas input 0 of step 11836 have datatype=18 WARNING : type of output 3 of step 11838 doesn't seem to be define in the database( WARNING : type of input 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 2 of step 11838 doesn't seem to be define in the database( WARNING : output 1 of step 10384 have datatype=7 whereas input 2 of step 11838 have datatype=None WARNING : output 0 of step 10390 have datatype=1 whereas input 0 of step 10384 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=23265368 AND mptpi.`type`=4213 To do Qualite : 0.09688064980876268 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 10383 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 10391 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10392 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 10387 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 10387 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 10390 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 10390 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11838 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11838 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 10386 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 10385 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 10385 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 10393 send_mail_cod have less outputs used (0) than in the step definition (1) : some outputs may be not used ! Step 10430 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 3 of step 10385 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 10383 doesn't seem to be define in the database( WARNING : type of input 2 of step 10387 doesn't seem to be define in the database( WARNING : output 1 of step 10383 have datatype=2 whereas input 1 of step 10390 have datatype=7 WARNING : type of output 2 of step 10390 doesn't seem to be define in the database( WARNING : type of input 1 of step 10384 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 10386 have datatype=10 whereas input 3 of step 10393 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10391 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of output 1 of step 10392 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of input 5 of step 10393 doesn't seem to be define in the database( WARNING : output 0 of step 11836 have datatype=11 whereas input 5 of step 10393 have datatype=None WARNING : output 0 of step 10386 have datatype=10 whereas input 0 of step 11836 have datatype=18 WARNING : type of output 3 of step 11838 doesn't seem to be define in the database( WARNING : type of input 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 2 of step 11838 doesn't seem to be define in the database( WARNING : output 1 of step 10384 have datatype=7 whereas input 2 of step 11838 have datatype=None WARNING : output 0 of step 10390 have datatype=1 whereas input 0 of step 10384 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=23267379 AND mptpi.`type`=4213 To do Qualite : 0.09668143700212802 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 10383 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 10391 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10392 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 10387 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 10387 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 10390 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 10390 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 10384 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11838 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11838 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 10386 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 10385 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 10385 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 10393 send_mail_cod have less outputs used (0) than in the step definition (1) : some outputs may be not used ! Step 10430 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 3 of step 10385 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 10383 doesn't seem to be define in the database( WARNING : type of input 2 of step 10387 doesn't seem to be define in the database( WARNING : output 1 of step 10383 have datatype=2 whereas input 1 of step 10390 have datatype=7 WARNING : type of output 2 of step 10390 doesn't seem to be define in the database( WARNING : type of input 1 of step 10384 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 10386 have datatype=10 whereas input 3 of step 10393 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 10391 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of output 1 of step 10392 doesn't seem to be define in the database( WARNING : type of input 3 of step 10387 doesn't seem to be define in the database( WARNING : type of input 5 of step 10393 doesn't seem to be define in the database( WARNING : output 0 of step 11836 have datatype=11 whereas input 5 of step 10393 have datatype=None WARNING : output 0 of step 10386 have datatype=10 whereas input 0 of step 11836 have datatype=18 WARNING : type of output 3 of step 11838 doesn't seem to be define in the database( WARNING : type of input 1 of step 10386 doesn't seem to be define in the database( WARNING : type of input 2 of step 11838 doesn't seem to be define in the database( WARNING : output 1 of step 10384 have datatype=7 whereas input 2 of step 11838 have datatype=None WARNING : output 0 of step 10390 have datatype=1 whereas input 0 of step 10384 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=23270896 AND mptpi.`type`=4213 To do elapsed_time : count_nb_balles_and_create_portfolio 5.537090301513672 # DISPLAY ALL COLLECTED DATA : {'22052025': {'nb_upload': 253, 'nb_taggue_class': 148, 'nb_taggue_densite': 148, 'nb_descriptors': 147}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1360680983, 1360680934, 1360680928, 1360680923, 1360680768, 1360680692, 1360680643, 1360680564, 1360680482, 1360680439, 1360680322, 1360679838, 1360679325, 1360679084, 1360678804, 1360678744, 1360678660, 1360678651, 1360678109, 1360675091, 1360668647, 1360637838, 1360627637, 1360625315, 1360622183, 1360605718, 1360603801, 1360595420, 1360594525, 1360593628, 1360593574, 1360593564, 1360593559, 1360593555, 1360593552, 1360593548, 1360591908, 1360591858, 1360591812, 1360591772, 1360591732, 1360591694, 1360589255, 1360589200, 1360589158, 1360589111, 1360589072, 1360589041, 1360587330, 1360587292, 1360587253, 1360587198, 1360587141, 1360587101, 1360585163, 1360585156, 1360585152, 1360585145, 1360585140, 1360585137, 1360583146, 1360583143, 1360583139, 1360583134, 1360583128, 1360583116, 1360581002, 1360580970, 1360580941, 1360580913, 1360580882, 1360580855, 1360579999, 1360579998, 1360579774, 1360579768, 1360579745, 1360579727, 1360579713, 1360579677, 1360579145, 1360579112, 1360579084, 1360579047, 1360579019, 1360579011, 1360578474, 1360578468, 1360578462, 1360578455, 1360578450, 1360578447, 1360577279, 1360577244, 1360577233, 1360577228, 1360577224, 1360577219, 1360574569, 1360572202] Looping around the photos to save general results len do output : 1 /23234922Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360680983', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360680934', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360680928', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360680923', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360680768', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360680692', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360680643', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360680564', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360680482', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360680439', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360680322', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360679838', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360679325', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360679084', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360678804', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360678744', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360678660', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360678651', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360678109', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360675091', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360668647', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360637838', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360627637', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360625315', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360622183', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360605718', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360603801', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360595420', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360594525', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360593628', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360593574', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360593564', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360593559', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360593555', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360593552', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360593548', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360591908', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360591858', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360591812', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360591772', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360591732', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360591694', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360589255', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360589200', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360589158', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360589111', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360589072', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360589041', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360587330', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360587292', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360587253', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360587198', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360587141', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360587101', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360585163', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360585156', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360585152', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360585145', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360585140', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360585137', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360583146', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360583143', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360583139', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360583134', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360583128', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360583116', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360581002', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360580970', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360580941', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360580913', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360580882', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360580855', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360579999', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360579998', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360579774', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360579768', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360579745', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360579727', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360579713', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360579677', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360579145', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360579112', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360579084', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360579047', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360579019', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360579011', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360578474', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360578468', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360578462', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360578455', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360578450', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360578447', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360577279', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360577244', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360577233', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360577228', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360577224', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360577219', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360574569', None, None, None, None, None, '2944292') ('3142', None, None, None, None, None, None, None, '2944292') ('3142', '23234922', '1360572202', None, None, None, None, None, '2944292') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 101 time used for this insertion : 0.031213998794555664 save_final save missing photos in datou_result : time spend for datou_step_exec : 16.082453727722168 time spend to save output : 0.03201174736022949 total time spend for step 1 : 16.114465475082397 caffe_path_current : About to save ! 2 After save, about to update current ! ret : 2 len(input) + len(total_photo_id_missing) : 1 set_done_treatment 1.52user 1.09system 0:19.34elapsed 13%CPU (0avgtext+0avgdata 104392maxresident)k 11552inputs+1424outputs (54major+49684minor)pagefaults 0swaps