python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 20 -a ' -a 4876' -s datou_current_4876 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 319398 load datou : 4876 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 2745>1000 please execute split_portfolio.py -i 22042798 -l 1000 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 4876, datou_cur_ids : ['2726440'] with mtr_portfolio_ids : ['22042798'] and first list_photo_ids : [] new path : /proc/319398/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : split_time_score over limit max, limiting to limit_max 100 list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB time to download the photos : 0.02940201759338379 About to test input to load Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 1 step1:split_time_score Fri Apr 4 12:52:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec begin split time score 2022-04-13 10:29:59 0 TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 3379, 'mtr_user_id': 31, 'name': 'learn_classif_flux_maj_generique_effnet_v2_s_02062022', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'aluminium,ela,film_pedb,flux_dev,jrm,pcm,pcnc,pehd_pp,pet_clair,refus,tapis_vide', 'svm_portfolios_learning': '5515864,5515840,5515844,5515850,6244400,6237996,6237998,5515847,5515841,5515868,5515866', 'photo_hashtag_type': 4374, 'photo_desc_type': 5680, 'type_classification': 'tf_classification2', 'hashtag_id_list': '493546845,492741797,2107760237,2107760238,495916461,560181804,1284539308,2107760239,2107755846,538914404,2107748999'}] thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('00', 277), ('01', 132), ('02', 277), ('03', 269), ('04', 55), ('05', 262), ('06', 292), ('07', 277), ('08', 301), ('09', 129), ('10', 221), ('11', 210), ('12', 43)) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {1: 2676, 2: 1} 04042025 22042798 Nombre de photos uploadées : 301 / 23040 (1%) 04042025 22042798 Nombre de photos taguées (types de déchets): 2677 / 301 (889%) 04042025 22042798 Nombre de photos taguées (volume) : 2677 / 301 (889%) elapsed_time : load_data_split_time_score 1.6689300537109375e-06 elapsed_time : order_list_meta_photo_and_scores 0.0008304119110107422 ???? elapsed_time : fill_and_build_computed_from_old_data 0.05260181427001953 elapsed_time : insert_dashboard_record_day_entry 0.03052830696105957 Creating list_photo_by_hashtags Hashtag is None Hashtag is None Hashtag is None Hashtag is None elapsed_time : list_photo_by_hashtags 0.008840084075927734 photos_removed : len 0 elapsed_time : load_duplicate_info 0.011833906173706055 ***** BEGIN SPLIT TIME ***** True 1743724800.0 `1743756533.009129 1743756300.0 `1743756533.009129 1743756300.0 `1743756533.009129 1743756300.0 `1743756544.010159 1743756300.0 `1743756554.010139 1743756300.0 `1743756584.010135 1743756300.0 `1743756594.010133 1743756300.0 `1743756603.009122 1743756300.0 `1743756624.010133 1743756300.0 `1743756634.010134 1743756300.0 `1743756644.010133 1743756300.0 `1743756714.010133 1743756300.0 `1743756744.010134 1743756300.0 `1743756753.009122 1743756300.0 `1743756763.010133 1743756300.0 `1743756774.010135 1743756300.0 `1743756784.010144 1743756300.0 `1743756794.010134 1743756300.0 `1743756804.010134 1743756300.0 `1743756814.010133 1743756300.0 `1743756823.009124 1743756300.0 `1743756833.010134 1743756300.0 `1743756833.010134 1743756300.0 `1743756844.010144 1743756300.0 `1743756844.010144 1743756300.0 `1743756854.01019 1743756300.0 `1743756854.01019 1743756300.0 `1743756864.010135 1743756300.0 `1743756864.010135 1743756300.0 `1743756874.010135 1743756300.0 `1743756924.010141 1743756300.0 `1743756944.010139 1743756300.0 `1743756944.010139 1743756300.0 `1743757004.010136 1743756300.0 `1743757043.010138 1743756300.0 `1743757054.010133 1743756300.0 `1743757064.010134 1743756300.0 `1743757074.010135 1743756300.0 `1743757084.010163 1743756300.0 `1743757094.010136 1743756300.0 `1743757104.010133 1743756300.0 `1743757113.009123 1743756300.0 `1743757123.010138 1743756300.0 `1743757134.010137 1743756300.0 `1743757154.011008 1743756300.0 `1743757183.010134 1743756300.0 `1743757193.010136 1743756300.0 `1743757193.010136 1743756300.0 `1743757204.010139 1743757200.0 `1743757204.010139 1743757200.0 `1743757214.010137 1743757200.0 `1743757214.010137 1743757200.0 `1743757224.010137 1743757200.0 `1743757224.010137 1743757200.0 `1743757233.009127 1743757200.0 `1743757233.009127 1743757200.0 `1743757243.010135 1743757200.0 `1743757243.010135 1743757200.0 `1743757253.010135 1743757200.0 `1743757263.010143 1743757200.0 `1743757303.009122 1743757200.0 `1743757313.010134 1743757200.0 `1743757323.01016 1743757200.0 `1743757333.010136 1743757200.0 `1743757344.010132 1743757200.0 `1743757354.010133 1743757200.0 `1743757364.010134 1743757200.0 `1743757374.010133 1743757200.0 `1743757383.009129 1743757200.0 `1743757393.010193 1743757200.0 `1743757403.010133 1743757200.0 `1743757424.010134 1743757200.0 `1743757434.010133 1743757200.0 `1743757444.01015 1743757200.0 `1743757453.009126 1743757200.0 `1743757463.010137 1743757200.0 `1743757473.010138 1743757200.0 `1743757483.010136 1743757200.0 `1743757494.010138 1743757200.0 `1743757504.010151 1743757200.0 `1743757514.010138 1743757200.0 `1743757523.009126 1743757200.0 `1743757533.010136 1743757200.0 `1743757574.010133 1743757200.0 `1743757584.010133 1743757200.0 `1743757593.009124 1743757200.0 `1743757603.010135 1743757200.0 `1743757613.010138 1743757200.0 `1743757623.010138 1743757200.0 `1743757634.010137 1743757200.0 `1743757644.010143 1743757200.0 `1743757654.010138 1743757200.0 `1743757664.010138 1743757200.0 `1743757673.009128 1743757200.0 `1743757683.010145 1743757200.0 `1743757693.010133 1743757200.0 `1743757724.010133 1743757200.0 `1743757734.010133 1743757200.0 `1743757743.009128 1743757200.0 `1743757753.010141 1743757200.0 `1743757763.010143 1743757200.0 `1743757773.010135 1743757200.0 `1743757784.010134 1743757200.0 `1743757794.010136 1743757200.0 `1743757804.010151 1743757200.0 `1743757813.009129 1743757200.0 `1743757833.010133 1743757200.0 `1743757843.010135 1743757200.0 `1743757854.010136 1743757200.0 `1743757854.010136 1743757200.0 `1743757864.010149 1743757200.0 `1743757864.010149 1743757200.0 `1743757874.010138 1743757200.0 `1743757874.010138 1743757200.0 `1743757883.009124 1743757200.0 `1743757883.009124 1743757200.0 `1743757893.010134 1743757200.0 `1743757893.010134 1743757200.0 `1743757903.010134 1743757200.0 `1743757903.010134 1743757200.0 `1743757913.010138 1743757200.0 `1743757924.010156 1743757200.0 `1743757944.010135 1743757200.0 `1743757954.010133 1743757200.0 `1743757963.009125 1743757200.0 `1743757973.010136 1743757200.0 `1743757983.01014 1743757200.0 `1743757993.010136 1743757200.0 `1743758004.010141 1743757200.0 `1743758014.010137 1743757200.0 `1743758024.010135 1743757200.0 `1743758033.009124 1743757200.0 `1743758063.010133 1743757200.0 `1743758074.010133 1743757200.0 `1743758084.010135 1743757200.0 `1743758094.010135 1743757200.0 `1743758103.009137 1743758100.0 `1743758113.010141 1743758100.0 `1743758123.010144 1743758100.0 `1743758144.01014 1743758100.0 `1743758154.010136 1743758100.0 `1743758164.010138 1743758100.0 `1743758174.010136 1743758100.0 `1743758183.009124 1743758100.0 `1743758203.010135 1743758100.0 `1743758213.010136 1743758100.0 `1743758224.010136 1743758100.0 `1743758244.010135 1743758100.0 `1743758253.009124 1743758100.0 `1743758304.010132 1743758100.0 `1743758314.010135 1743758100.0 `1743758323.009124 1743758100.0 `1743758333.010135 1743758100.0 `1743758343.010136 1743758100.0 `1743758353.010136 1743758100.0 `1743758364.010195 1743758100.0 `1743758374.010134 1743758100.0 `1743758384.010134 1743758100.0 `1743758393.009126 1743758100.0 `1743758403.01016 1743758100.0 `1743758444.010136 1743758100.0 `1743758454.010137 1743758100.0 `1743758473.010138 1743758100.0 `1743758483.010139 1743758100.0 `1743758493.010134 1743758100.0 `1743758503.010134 1743758100.0 `1743758514.010134 1743758100.0 `1743758543.00913 1743758100.0 `1743758553.010134 1743758100.0 `1743758563.010136 1743758100.0 `1743758584.010163 1743758100.0 `1743758594.010135 1743758100.0 `1743758613.009124 1743758100.0 `1743758623.010138 1743758100.0 `1743758633.010141 1743758100.0 `1743758643.010145 1743758100.0 `1743758664.010134 1743758100.0 `1743758674.010133 1743758100.0 `1743758683.009123 1743758100.0 `1743758693.010138 1743758100.0 `1743758703.010141 1743758100.0 `1743758713.010133 1743758100.0 `1743758723.010133 1743758100.0 `1743758734.010136 1743758100.0 `1743758773.010136 1743758100.0 `1743758783.010134 1743758100.0 `1743758804.010138 1743758100.0 `1743758814.010135 1743758100.0 `1743758824.010163 1743758100.0 `1743758833.009126 1743758100.0 `1743758843.010138 1743758100.0 `1743758853.010133 1743758100.0 `1743758863.010135 1743758100.0 `1743758874.010137 1743758100.0 `1743758884.010143 1743758100.0 `1743758894.010136 1743758100.0 `1743758923.010134 1743758100.0 `1743758933.010132 1743758100.0 `1743758943.010134 1743758100.0 `1743758954.010136 1743758100.0 `1743758964.010134 1743758100.0 `1743758974.010135 1743758100.0 `1743758983.009126 1743758100.0 `1743758993.010136 1743758100.0 `1743759003.010141 1743759000.0 `1743759013.010137 1743759000.0 `1743759024.010133 1743759000.0 `1743759044.010133 1743759000.0 `1743759053.009127 1743759000.0 `1743759063.010146 1743759000.0 `1743759073.010138 1743759000.0 `1743759083.010133 1743759000.0 `1743759094.010133 1743759000.0 `1743759104.010135 1743759000.0 `1743759114.010135 1743759000.0 `1743759143.010133 1743759000.0 `1743759153.010134 1743759000.0 `1743759163.010135 1743759000.0 `1743759174.010134 1743759000.0 `1743759184.010152 1743759000.0 `1743759194.010136 1743759000.0 `1743759203.009126 1743759000.0 `1743759223.010143 1743759000.0 `1743759254.010134 1743759000.0 `1743759293.010136 1743759000.0 `1743759314.010136 1743759000.0 `1743759334.010144 1743759000.0 `1743759343.009129 1743759000.0 `1743759353.010137 1743759000.0 `1743759394.010143 1743759000.0 `1743759404.010143 1743759000.0 `1743759414.01014 1743759000.0 `1743759423.009137 1743759000.0 `1743759433.010141 1743759000.0 `1743759443.010141 1743759000.0 `1743759453.010141 1743759000.0 `1743759503.010134 1743759000.0 `1743759513.010137 1743759000.0 `1743759544.010145 1743759000.0 `1743759554.010138 1743759000.0 `1743759563.009124 1743759000.0 `1743759573.010141 1743759000.0 `1743759583.01014 1743759000.0 `1743759593.010134 1743759000.0 `1743759604.010155 1743759000.0 `1743759614.010137 1743759000.0 `1743759624.010135 1743759000.0 `1743760493.009124 1743759900.0 `1743760503.010148 1743759900.0 `1743760523.010137 1743759900.0 `1743760534.010142 1743759900.0 `1743760544.010142 1743759900.0 `1743760573.01014 1743759900.0 `1743760633.009128 1743759900.0 `1743760643.01014 1743759900.0 `1743760673.010136 1743759900.0 `1743760684.010136 1743759900.0 `1743760694.010135 1743759900.0 `1743760925.010159 1743760800.0 `1743760935.010134 1743760800.0 `1743760964.010139 1743760800.0 `1743760974.010136 1743760800.0 `1743760985.010152 1743760800.0 `1743760995.010134 1743760800.0 `1743761005.010137 1743760800.0 `1743761015.010134 1743760800.0 `1743761024.009125 1743760800.0 `1743761034.010136 1743760800.0 `1743761044.010149 1743760800.0 `1743761054.010138 1743760800.0 `1743761065.010143 1743760800.0 `1743761075.010135 1743760800.0 `1743761094.009125 1743760800.0 `1743761104.010151 1743760800.0 `1743761114.010133 1743760800.0 `1743761124.010134 1743760800.0 `1743761135.010135 1743760800.0 `1743761145.010135 1743760800.0 `1743761155.010136 1743760800.0 `1743761164.009137 1743760800.0 `1743761174.010288 1743760800.0 `1743761184.010134 1743760800.0 `1743761195.010135 1743760800.0 `1743761205.010134 1743760800.0 `1743761215.010135 1743760800.0 `1743761224.009139 1743760800.0 `1743761234.010135 1743760800.0 `1743761244.010132 1743760800.0 `1743761264.010134 1743760800.0 `1743761275.010135 1743760800.0 `1743761285.010166 1743760800.0 `1743761295.010135 1743760800.0 `1743761314.010135 1743760800.0 `1743761324.010138 1743760800.0 `1743761345.010146 1743760800.0 `1743761365.010135 1743760800.0 `1743761384.010139 1743760800.0 `1743761404.010139 1743760800.0 `1743761415.010134 1743760800.0 `1743761454.010134 1743760800.0 `1743761814.009125 1743761700.0 list printed: [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], [48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58,