python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 20 -a ' -a 4189' -s datou_current_4189 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3505002 load datou : 4189 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 list_input_json: {} Date 2026-04-08 10:30:29.187572 Current got : datou_id : 4189, datou_cur_ids : ['4372757'] with mtr_portfolio_ids : ['30953718'] and first list_photo_ids : [] new path : /proc/3505002/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : split_time_score over limit max, limiting to limit_max 100 list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB time to download the photos : 0.02179265022277832 About to test input to load Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 1 step1:split_time_score Wed Apr 8 10:30:29 2026 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec begin split time score 2022-04-13 10:29:59 0 TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 3379, 'mtr_user_id': 31, 'name': 'learn_classif_flux_maj_generique_effnet_v2_s_02062022', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'aluminium,ela,film_pedb,flux_dev,jrm,pcm,pcnc,pehd_pp,pet_clair,refus,tapis_vide', 'svm_portfolios_learning': '5515864,5515840,5515844,5515850,6244400,6237996,6237998,5515847,5515841,5515868,5515866', 'photo_hashtag_type': 4374, 'photo_desc_type': 5680, 'type_classification': 'tf_classification2', 'hashtag_id_list': '493546845,492741797,2107760237,2107760238,495916461,560181804,1284539308,2107760239,2107755846,538914404,2107748999'}] thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('10', 149),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {1: 6} L08042026 30953718 Nombre de photos uploadées : 149 / 23040 (0%) 08042026 30953718 Nombre de photos taguées (types de déchets): 6 / 149 (4%) 08042026 30953718 Nombre de photos taguées (volume) : 6 / 149 (4%) elapsed_time : load_data_split_time_score 2.1457672119140625e-06 elapsed_time : order_list_meta_photo_and_scores 1.3828277587890625e-05 LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.007982969284057617 LCatched exception ! Connect or reconnect ! Catched exception ! Connect or reconnect ! elapsed_time : insert_dashboard_record_day_entry 0.22201895713806152 LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL Creating list_photo_by_hashtags Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None elapsed_time : list_photo_by_hashtags 0.016139745712280273 Creating list_photo_total elapsed_time : select_descriptors 0.7804265022277832 08042026 30953718 Nombre de photos avec descriptors (type 5680) : 6 / 149 (4%) Missing descriptors for photos 0 and 1415257221 L0:00:00|ON:Missing descriptors for photos 1415257221 and 1415257222 LMissing descriptors for photos 1415257222 and 1415257223 LMissing descriptors for photos 1415257223 and 1415257224 LMissing descriptors for photos 1415257224 and 1415257225 LMissing descriptors for photos 1415257225 and 1415257227 LMissing descriptors for photos 1415257227 and 1415257228 LMissing descriptors for photos 1415257228 and 1415257229 LMissing descriptors for photos 1415257229 and 1415257230 LMissing descriptors for photos 1415257230 and 1415257231 LMissing descriptors for photos 1415257231 and 1415257232 LMissing descriptors for photos 1415257232 and 1415257233 LMissing descriptors for photos 1415257233 and 1415257234 LMissing descriptors for photos 1415257234 and 1415257236 LMissing descriptors for photos 1415257236 and 1415257237 LMissing descriptors for photos 1415257237 and 1415257238 LMissing descriptors for photos 1415257238 and 1415257239 LMissing descriptors for photos 1415257239 and 1415257242 LMissing descriptors for photos 1415257242 and 1415257243 LMissing descriptors for photos 1415257243 and 1415257244 LMissing descriptors for photos 1415257244 and 1415257245 LMissing descriptors for photos 1415257245 and 1415257246 LMissing descriptors for photos 1415257246 and 1415257386 LMissing descriptors for photos 1415257386 and 1415258139 LMissing descriptors for photos 1415258139 and 1415258141 LMissing descriptors for photos 1415258141 and 1415258142 LMissing descriptors for photos 1415258142 and 1415258175 LMissing descriptors for photos 1415258175 and 1415258235 LMissing descriptors for photos 1415258235 and 1415258236 LMissing descriptors for photos 1415258236 and 1415258336 LMissing descriptors for photos 1415258336 and 1415258338 LMissing descriptors for photos 1415258338 and 1415258340 LMissing descriptors for photos 1415258340 and 1415258341 LMissing descriptors for photos 1415258341 and 1415258343 LMissing descriptors for photos 1415258343 and 1415258344 LMissing descriptors for photos 1415258344 and 1415258348 LMissing descriptors for photos 1415258348 and 1415258351 LMissing descriptors for photos 1415258351 and 1415258354 LMissing descriptors for photos 1415258354 and 1415258355 LMissing descriptors for photos 1415258355 and 1415258356 LMissing descriptors for photos 1415258356 and 1415258357 LERROR : Hum hum, what can we do for different size of descriptors (ignore the difference ) : 0 vs 1280 photo_id : 1415259006 photo_id_prec : 1415258357 LLLLLLERROR : Hum hum, what can we do for different size of descriptors (ignore the difference ) : 1280 vs 0 photo_id : 1415259235 photo_id_prec : 1415259058 LMissing descriptors for photos 1415259235 and 1415259283 LMissing descriptors for photos 1415259283 and 1415259286 LMissing descriptors for photos 1415259286 and 1415259287 LMissing descriptors for photos 1415259287 and 1415259288 LMissing descriptors for photos 1415259288 and 1415259289 LMissing descriptors for photos 1415259289 and 1415259290 LMissing descriptors for photos 1415259290 and 1415259291 LMissing descriptors for photos 1415259291 and 1415259292 LMissing descriptors for photos 1415259292 and 1415259293 LMissing descriptors for photos 1415259293 and 1415259295 LMissing descriptors for photos 1415259295 and 1415259296 LMissing descriptors for photos 1415259296 and 1415259297 LMissing descriptors for photos 1415259297 and 1415259298 LMissing descriptors for photos 1415259298 and 1415259300 LMissing descriptors for photos 1415259300 and 1415259301 LMissing descriptors for photos 1415259301 and 1415259303 LMissing descriptors for photos 1415259303 and 1415259577 LMissing descriptors for photos 1415259577 and 1415259860 LMissing descriptors for photos 1415259860 and 1415259861 LMissing descriptors for photos 1415259861 and 1415259896 LMissing descriptors for photos 1415259896 and 1415259926 LMissing descriptors for photos 1415259926 and 1415259927 LMissing descriptors for photos 1415259927 and 1415259928 LMissing descriptors for photos 1415259928 and 1415259931 LMissing descriptors for photos 1415259931 and 1415259933 LMissing descriptors for photos 1415259933 and 1415259934 LMissing descriptors for photos 1415259934 and 1415259935 LMissing descriptors for photos 1415259935 and 1415259937 LMissing descriptors for photos 1415259937 and 1415259938 LMissing descriptors for photos 1415259938 and 1415259942 LMissing descriptors for photos 1415259942 and 1415259943 LMissing descriptors for photos 1415259943 and 1415259944 LMissing descriptors for photos 1415259944 and 1415259945 LMissing descriptors for photos 1415259945 and 1415259946 LMissing descriptors for photos 1415259946 and 1415259947 LMissing descriptors for photos 1415259947 and 1415259948 LMissing descriptors for photos 1415259948 and 1415259949 LMissing descriptors for photos 1415259949 and 1415259950 LMissing descriptors for photos 1415259950 and 1415259951 LMissing descriptors for photos 1415259951 and 1415259952 LMissing descriptors for photos 1415259952 and 1415259953 LMissing descriptors for photos 1415259953 and 1415259955 LMissing descriptors for photos 1415259955 and 1415259956 LMissing descriptors for photos 1415259956 and 1415259957 LMissing descriptors for photos 1415259957 and 1415259958 LMissing descriptors for photos 1415259958 and 1415259959 LMissing descriptors for photos 1415259959 and 1415259961 LMissing descriptors for photos 1415259961 and 1415259966 LMissing descriptors for photos 1415259966 and 1415259968 LMissing descriptors for photos 1415259968 and 1415259971 LMissing descriptors for photos 1415259971 and 1415259972 LMissing descriptors for photos 1415259972 and 1415259973 LMissing descriptors for photos 1415259973 and 1415259974 LMissing descriptors for photos 1415259974 and 1415259977 LMissing descriptors for photos 1415259977 and 1415259978 LMissing descriptors for photos 1415259978 and 1415259979 LMissing descriptors for photos 1415259979 and 1415259980 LMissing descriptors for photos 1415259980 and 1415259981 LMissing descriptors for photos 1415259981 and 1415259982 LMissing descriptors for photos 1415259982 and 1415259988 LMissing descriptors for photos 1415259988 and 1415259990 LMissing descriptors for photos 1415259990 and 1415259991 LMissing descriptors for photos 1415259991 and 1415259992 LMissing descriptors for photos 1415259992 and 1415259993 LMissing descriptors for photos 1415259993 and 1415259994 LMissing descriptors for photos 1415259994 and 1415260000 LMissing descriptors for photos 1415260000 and 1415260002 LMissing descriptors for photos 1415260002 and 1415260004 LMissing descriptors for photos 1415260004 and 1415260005 LMissing descriptors for photos 1415260005 and 1415260006 LMissing descriptors for photos 1415260006 and 1415260007 LMissing descriptors for photos 1415260007 and 1415260009 LMissing descriptors for photos 1415260009 and 1415260010 LMissing descriptors for photos 1415260010 and 1415260011 LMissing descriptors for photos 1415260011 and 1415260012 LMissing descriptors for photos 1415260012 and 1415260013 LMissing descriptors for photos 1415260013 and 1415260015 LMissing descriptors for photos 1415260015 and 1415260017 LMissing descriptors for photos 1415260017 and 1415260018 LMissing descriptors for photos 1415260018 and 1415260019 LMissing descriptors for photos 1415260019 and 1415260020 LMissing descriptors for photos 1415260020 and 1415260021 LMissing descriptors for photos 1415260021 and 1415260023 LMissing descriptors for photos 1415260023 and 1415260024 LMissing descriptors for photos 1415260024 and 1415260025 LMissing descriptors for photos 1415260025 and 1415260026 LMissing descriptors for photos 1415260026 and 1415260028 LMissing descriptors for photos 1415260028 and 1415260030 LMissing descriptors for photos 1415260030 and 1415260031 LMissing descriptors for photos 1415260031 and 1415260032 LMissing descriptors for photos 1415260032 and 1415260033 LMissing descriptors for photos 1415260033 and 1415260034 LMissing descriptors for photos 1415260034 and 1415260035 LMissing descriptors for photos 1415260035 and 1415260036 LMissing descriptors for photos 1415260036 and 1415260037 LMissing descriptors for photos 1415260037 and 1415260039 LMissing descriptors for photos 1415260039 and 1415260040 LMissing descriptors for photos 1415260040 and 1415260041 LMissing descriptors for photos 1415260041 and 1415260043 LMissing descriptors for photos 1415260043 and 1415260045 LMissing descriptors for photos 1415260045 and 1415260046 L 08042026 Removing 0 photos because of the 'same image' condition Total on : 0 Total off : 0.0 list_time_off Warning in study_and_display_distrib_list : min=max : 0.0 0.0 dist_desc begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 149 time used for this insertion : 0.05650687217712402 photos_removed : len 0 elapsed_time : remove_photo_duplicate 0.10295581817626953 To do, maybe not at the correct place ! .L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.L.Lforce hashtag to JRM elapsed_time : CREATE_PORT_BATCH_BY_HOUR 0.009939908981323242 NUMBER BATCH : 2 list_ponderation used : [1e-05, 1e-05, 1e-05, 1e-05, 1e-05] , list_hashtag_class_create_as_list : ['jrm'] LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info LLLLLLERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info ERROR missing amount info LLresult_one_balle_Type_JRM:{'day': '08042026', 'map_nb_amount': {0: 1, 1: 4, 2: 0, 3: 0, 4: 0}, 'map_time_amount': {0: 0, 1: 0, 2: 0, 3: 0, 4: 0}, 'duration': 839.7116029262543, 'nb_balles_papier': 6e-05, 'begin_time_port': 'image_08042026_10_00_02_865602_0000.jpg'} Production hashtag (incorrect ponderation at 20-10-18) : 6e-05 LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLWe filter photos on hashtag condition ! We have rejected 0 photos because of the batch_size condition ! NUMBER BATCH list_of_portfolios_to_create : 1 list_same_port_ids : [] # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 12489 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 12499 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12500 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 12492 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 12492 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! WARNING : number of outputs for step 12493 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12494 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 12502 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 12502 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 12496 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 12495 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 12495 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 2 of step 12492 doesn't seem to be define in the database( WARNING : output 1 of step 12489 have datatype=2 whereas input 1 of step 12493 have datatype=7 WARNING : type of output 2 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12494 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 12493 doesn't seem to be define in the database( WARNING : type of input 1 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12496 doesn't seem to be define in the database( WARNING : type of input 3 of step 12495 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12500 doesn't seem to be define in the database( WARNING : type of output 2 of step 12489 doesn't seem to be define in the database( WARNING : type of input 1 of step 12499 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 12496 have datatype=10 whereas input 3 of step 12498 have datatype=6 WARNING : type of input 5 of step 12498 doesn't seem to be define in the database( WARNING : output 0 of step 12501 have datatype=11 whereas input 5 of step 12498 have datatype=None WARNING : output 0 of step 12496 have datatype=10 whereas input 0 of step 12501 have datatype=18 WARNING : type of input 2 of step 12502 doesn't seem to be define in the database( WARNING : output 1 of step 12494 have datatype=7 whereas input 2 of step 12502 have datatype=None WARNING : type of output 3 of step 12502 doesn't seem to be define in the database( WARNING : type of input 2 of step 12496 doesn't seem to be define in the database( WARNING : type of output 1 of step 12499 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : type of output 1 of step 12500 doesn't seem to be define in the database( WARNING : type of input 3 of step 12492 doesn't seem to be define in the database( WARNING : output 0 of step 12493 have datatype=1 whereas input 0 of step 12494 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=30953845 AND mptpi.`type`=3726 To do elapsed_time : count_nb_balles_and_create_portfolio 2.747927665710449 # DISPLAY ALL COLLECTED DATA : {'08042026': {'nb_upload': 149, 'nb_taggue_class': 6, 'nb_taggue_densite': 6, 'nb_descriptors': 6}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1415260046, 1415260045, 1415260043, 1415260041, 1415260040, 1415260039, 1415260037, 1415260036, 1415260035, 1415260034, 1415260033, 1415260032, 1415260031, 1415260030, 1415260028, 1415260026, 1415260025, 1415260024, 1415260023, 1415260021, 1415260020, 1415260019, 1415260018, 1415260017, 1415260015, 1415260013, 1415260012, 1415260011, 1415260010, 1415260009, 1415260007, 1415260006, 1415260005, 1415260004, 1415260002, 1415260000, 1415259994, 1415259993, 1415259992, 1415259991, 1415259990, 1415259988, 1415259982, 1415259981, 1415259980, 1415259979, 1415259978, 1415259977, 1415259974, 1415259973, 1415259972, 1415259971, 1415259968, 1415259966, 1415259961, 1415259959, 1415259958, 1415259957, 1415259956, 1415259955, 1415259953, 1415259952, 1415259951, 1415259950, 1415259949, 1415259948, 1415259947, 1415259946, 1415259945, 1415259944, 1415259943, 1415259942, 1415259938, 1415259937, 1415259935, 1415259934, 1415259933, 1415259931, 1415259928, 1415259927, 1415259926, 1415259896, 1415259861, 1415259860, 1415259577, 1415259303, 1415259301, 1415259300, 1415259298, 1415259297, 1415259296, 1415259295, 1415259293, 1415259292, 1415259291, 1415259290, 1415259289, 1415259288, 1415259287, 1415259286] Looping around the photos to save general results len do output : 1 /30953718Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260046', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260045', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260043', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260041', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260040', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260039', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260037', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260036', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260035', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260034', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260033', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260032', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260031', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260030', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260028', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260026', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260025', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260024', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260023', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260021', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260020', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260019', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260018', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260017', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260015', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260013', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260012', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260011', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260010', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260009', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260007', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260006', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260005', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260004', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260002', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415260000', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259994', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259993', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259992', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259991', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259990', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259988', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259982', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259981', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259980', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259979', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259978', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259977', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259974', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259973', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259972', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259971', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259968', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259966', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259961', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259959', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259958', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259957', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259956', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259955', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259953', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259952', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259951', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259950', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259949', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259948', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259947', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259946', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259945', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259944', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259943', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259942', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259938', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259937', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259935', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259934', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259933', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259931', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259928', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259927', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259926', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259896', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259861', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259860', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259577', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259303', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259301', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259300', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259298', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259297', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259296', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259295', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259293', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259292', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259291', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259290', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259289', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259288', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259287', None, None, None, None, None, '4372757') ('4189', None, None, None, None, None, None, None, '4372757') ('4189', '30953718', '1415259286', None, None, None, None, None, '4372757') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 101 time used for this insertion : 0.04867410659790039 save_final save missing photos in datou_result : time spend for datou_step_exec : 3.9858040809631348 time spend to save output : 0.049428462982177734 total time spend for step 1 : 4.0352325439453125 caffe_path_current : About to save ! 2 After save, about to update current ! ret : 2 len(input) + len(total_photo_id_missing) : 1 set_done_treatment 1.37user 0.61system 0:07.37elapsed 27%CPU (0avgtext+0avgdata 112380maxresident)k 144inputs+144outputs (5major+65513minor)pagefaults 0swaps