python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 20 -a ' -a 4876' -s datou_current_4876 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2421456 load datou : 4876 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 2894>1000 please execute split_portfolio.py -i 22087046 -l 1000 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 4876, datou_cur_ids : ['2730485'] with mtr_portfolio_ids : ['22087046'] and first list_photo_ids : [] new path : /proc/2421456/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : split_time_score over limit max, limiting to limit_max 100 list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB time to download the photos : 0.037601470947265625 About to test input to load Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 1 step1:split_time_score Mon Apr 7 21:34:27 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec begin split time score 2022-04-13 10:29:59 0 TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 3379, 'mtr_user_id': 31, 'name': 'learn_classif_flux_maj_generique_effnet_v2_s_02062022', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'aluminium,ela,film_pedb,flux_dev,jrm,pcm,pcnc,pehd_pp,pet_clair,refus,tapis_vide', 'svm_portfolios_learning': '5515864,5515840,5515844,5515850,6244400,6237996,6237998,5515847,5515841,5515868,5515866', 'photo_hashtag_type': 4374, 'photo_desc_type': 5680, 'type_classification': 'tf_classification2', 'hashtag_id_list': '493546845,492741797,2107760237,2107760238,495916461,560181804,1284539308,2107760239,2107755846,538914404,2107748999'}] thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('05', 60), ('06', 179), ('07', 223), ('08', 225), ('09', 96), ('10', 239), ('11', 235), ('12', 18), ('13', 26), ('14', 260), ('15', 270), ('16', 254), ('17', 141), ('18', 239), ('19', 241), ('20', 168), ('21', 20)) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {1: 2839} 07042025 22087046 Nombre de photos uploadées : 294 / 23040 (1%) 07042025 22087046 Nombre de photos taguées (types de déchets): 2839 / 294 (965%) 07042025 22087046 Nombre de photos taguées (volume) : 2839 / 294 (965%) elapsed_time : load_data_split_time_score 3.5762786865234375e-06 elapsed_time : order_list_meta_photo_and_scores 0.0010290145874023438 ????????? elapsed_time : fill_and_build_computed_from_old_data 0.015281438827514648 elapsed_time : insert_dashboard_record_day_entry 0.026669025421142578 Creating list_photo_by_hashtags Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None Hashtag is None elapsed_time : list_photo_by_hashtags 0.010885477066040039 photos_removed : len 0 elapsed_time : load_duplicate_info 0.013526439666748047 ***** BEGIN SPLIT TIME ***** True 1743984000.0 `1744047225.010139 1744047000.0 `1744047264.010135 1744047000.0 `1744047275.010137 1744047000.0 `1744047285.010139 1744047000.0 `1744047295.010139 1744047000.0 `1744047304.009134 1744047000.0 `1744047314.010137 1744047000.0 `1744047324.010135 1744047000.0 `1744047334.010141 1744047000.0 `1744047345.01014 1744047000.0 `1744047355.010138 1744047000.0 `1744047394.010132 1744047000.0 `1744047404.010134 1744047000.0 `1744047414.01014 1744047000.0 `1744047425.01017 1744047000.0 `1744047435.010135 1744047000.0 `1744047444.009126 1744047000.0 `1744047454.010143 1744047000.0 `1744047464.010137 1744047000.0 `1744047474.010136 1744047000.0 `1744047484.010142 1744047000.0 `1744047495.01014 1744047000.0 `1744047505.010136 1744047000.0 `1744047514.009136 1744047000.0 `1744047524.010133 1744047000.0 `1744047554.010134 1744047000.0 `1744047565.010134 1744047000.0 `1744047575.010135 1744047000.0 `1744047585.010139 1744047000.0 `1744047594.009127 1744047000.0 `1744047604.010137 1744047000.0 `1744047614.01014 1744047000.0 `1744047635.010139 1744047000.0 `1744047645.010138 1744047000.0 `1744047655.010137 1744047000.0 `1744047664.009133 1744047000.0 `1744047674.010136 1744047000.0 `1744047684.010133 1744047000.0 `1744047754.010141 1744047000.0 `1744047764.01014 1744047000.0 `1744047785.010152 1744047000.0 `1744047795.010137 1744047000.0 `1744047824.01014 1744047000.0 `1744047844.010144 1744047000.0 `1744047855.010135 1744047000.0 `1744047865.010134 1744047000.0 `1744047884.009126 1744047000.0 `1744047894.010135 1744047000.0 `1744047904.010153 1744047900.0 `1744047914.010137 1744047900.0 `1744047925.010136 1744047900.0 `1744047935.010136 1744047900.0 `1744047964.010153 1744047900.0 `1744047974.010133 1744047900.0 `1744047984.010144 1744047900.0 `1744047995.010137 1744047900.0 `1744048005.01014 1744047900.0 `1744048015.010137 1744047900.0 `1744048024.009139 1744047900.0 `1744048034.010138 1744047900.0 `1744048054.010135 1744047900.0 `1744048075.010136 1744047900.0 `1744048085.01014 1744047900.0 `1744048095.01014 1744047900.0 `1744048134.010136 1744047900.0 `1744048145.010153 1744047900.0 `1744048184.010139 1744047900.0 `1744048194.010136 1744047900.0 `1744048204.010138 1744047900.0 `1744048215.010137 1744047900.0 `1744048225.010136 1744047900.0 `1744048235.010139 1744047900.0 `1744048244.009126 1744047900.0 `1744048254.010136 1744047900.0 `1744048264.010154 1744047900.0 `1744048274.010138 1744047900.0 `1744048284.010136 1744047900.0 `1744048295.010134 1744047900.0 `1744048305.010135 1744047900.0 `1744048324.010155 1744047900.0 `1744048334.010135 1744047900.0 `1744048365.010134 1744047900.0 `1744048375.010141 1744047900.0 `1744048385.01015 1744047900.0 `1744048394.009124 1744047900.0 `1744048404.010136 1744047900.0 `1744048414.010136 1744047900.0 `1744048424.010137 1744047900.0 `1744048435.010136 1744047900.0 `1744048445.01016 1744047900.0 `1744048455.010136 1744047900.0 `1744048474.010135 1744047900.0 `1744048494.010139 1744047900.0 `1744048505.010151 1744047900.0 `1744048525.010138 1744047900.0 `1744048534.009135 1744047900.0 `1744048544.010144 1744047900.0 `1744048574.010135 1744047900.0 `1744048585.010132 1744047900.0 `1744048634.010133 1744047900.0 `1744048644.010136 1744047900.0 `1744048655.010136 1744047900.0 `1744048665.010137 1744047900.0 `1744048675.010136 1744047900.0 `1744048753.010026 1744047900.0 `1744048763.010149 1744047900.0 `1744048983.01014 1744048800.0 `1744048993.010142 1744048800.0 `1744049003.010145 1744048800.0 `1744049024.010141 1744048800.0 `1744049043.009132 1744048800.0 `1744049053.010138 1744048800.0 `1744049073.010147 1744048800.0 `1744049084.010136 1744048800.0 `1744049123.010133 1744048800.0 `1744049143.010135 1744048800.0 `1744049164.010163 1744048800.0 `1744049183.009125 1744048800.0 `1744049193.010141 1744048800.0 `1744049203.010143 1744048800.0 `1744049213.010137 1744048800.0 `1744049223.010152 1744048800.0 `1744049234.010135 1744048800.0 `1744049244.010139 1744048800.0 `1744049254.010136 1744048800.0 `1744049263.009126 1744048800.0 `1744049304.010134 1744048800.0 `1744049314.010134 1744048800.0 `1744049324.010137 1744048800.0 `1744049333.009127 1744048800.0 `1744049343.010157 1744048800.0 `1744049353.010143 1744048800.0 `1744049363.010141 1744048800.0 `1744049374.010146 1744048800.0 `1744049384.010143 1744048800.0 `1744049394.010137 1744048800.0 `1744049403.009141 1744048800.0 `1744049433.010133 1744048800.0 `1744049444.010133 1744048800.0 `1744049454.010135 1744048800.0 `1744049464.010145 1744048800.0 `1744049473.009127 1744048800.0 `1744049483.010136 1744048800.0 `1744049493.010138 1744048800.0 `1744049503.010137 1744048800.0 `1744049513.01014 1744048800.0 `1744049524.010165 1744048800.0 `1744049534.01014 1744048800.0 `1744049543.009126 1744048800.0 `1744049553.010136 1744048800.0 `1744049563.010135 1744048800.0 `1744049573.010139 1744048800.0 `1744049583.010155 1744048800.0 `1744049623.010133 1744048800.0 `1744049633.010137 1744048800.0 `1744049643.010159 1744048800.0 `1744049653.010136 1744048800.0 `1744049664.010135 1744048800.0 `1744049674.010137 1744048800.0 `1744049684.010138 1744048800.0 `1744049693.009129 1744048800.0 `1744049703.010159 1744049700.0 `1744049713.010141 1744049700.0 `1744049723.010142 1744049700.0 `1744049734.010134 1744049700.0 `1744049744.010135 1744049700.0 `1744049754.010137 1744049700.0 `1744049793.010133 1744049700.0 `1744049804.010133 1744049700.0 `1744049814.010134 1744049700.0 `1744049824.010152 1744049700.0 `1744049833.009125 1744049700.0 `1744049853.010145 1744049700.0 `1744049863.010141 1744049700.0 `1744049874.010136 1744049700.0 `1744049884.010159 1744049700.0 `1744049894.010142 1744049700.0 `1744049913.010135 1744049700.0 `1744049923.010138 1744049700.0 `1744049933.010133 1744049700.0 `1744049943.010148 1744049700.0 `1744049954.010141 1744049700.0 `1744049964.010134 1744049700.0 `1744049993.010135 1744049700.0 `1744050003.010146 1744049700.0 `1744050013.010138 1744049700.0 `1744050024.010133 1744049700.0 `1744050034.010136 1744049700.0 `1744050053.009126 1744049700.0 `1744050063.010153 1744049700.0 `1744050104.010138 1744049700.0 `1744050114.010138 1744049700.0 `1744050164.010141 1744049700.0 `1744050244.010156 1744049700.0 `1744050263.010137 1744049700.0 `1744050273.010139 1744049700.0 `1744050283.010139 1744049700.0 `1744050293.010141 1744049700.0 `1744050324.010136 1744049700.0 `1744050353.010139 1744049700.0 `1744050363.010163 1744049700.0 `1744050403.009127 1744049700.0 `1744050413.010133 1744049700.0 `1744050423.010148 1744049700.0 `1744050444.010137 1744049700.0 `1744050454.010141 1744049700.0 `1744050464.010139 1744049700.0 `1744050473.009127 1744049700.0 `1744050483.010162 1744049700.0 `1744050493.010137 1744049700.0 `1744050503.010135 1744049700.0 `1744050514.010136 1744049700.0 `1744050524.01014 1744049700.0 `1744050534.010133 1744049700.0 `1744050543.009142 1744049700.0 `1744050563.010139 1744049700.0 `1744050573.010133 1744049700.0 `1744050583.010134 1744049700.0 `1744050594.010137 1744049700.0 `1744050604.010146 1744050600.0 `1744050613.009126 1744050600.0 `1744050623.010137 1744050600.0 `1744050633.010134 1744050600.0 `1744050643.010137 1744050600.0 `1744050664.010142 1744050600.0 `1744050674.010139 1744050600.0 `1744050684.010142 1744050600.0 `1744050723.010137 1744050600.0 `1744050734.010137 1744050600.0 `1744050744.01014 1744050600.0 `1744050754.010136 1744050600.0 `1744050763.009128 1744050600.0 `1744050773.010137 1744050600.0 `1744050783.010149 1744050600.0 `1744050793.010139 1744050600.0 `1744050804.010139 1744050600.0 `1744050814.010133 1744050600.0 `1744050824.010136 1744050600.0 `1744050833.009127 1744050600.0 `1744050843.010139 1744050600.0 `1744050884.010136 1744050600.0 `1744050894.010143 1744050600.0 `1744050903.009163 1744050600.0 `1744050913.010149 1744050600.0 `1744050923.010208 1744050600.0 `1744050933.010151 1744050600.0 `1744050944.010144 1744050600.0 `1744050954.010141 1744050600.0 `1744050964.010178 1744050600.0 `1744050973.009129 1744050600.0 `1744050983.010138 1744050600.0 `1744050993.010136 1744050600.0 `1744051003.010137 1744050600.0 `1744051014.010136 1744050600.0 `1744051053.010135 1744050600.0 `1744051063.010133 1744050600.0 `1744051073.010147 1744050600.0 `1744051084.010153 1744050600.0 `1744051094.010142 1744050600.0 `1744051104.010137 1744050600.0 `1744051113.009127 1744050600.0 `1744051123.010138 1744050600.0 `1744051133.010136 1744050600.0 `1744051143.010166 1744050600.0 `1744051154.010138 1744050600.0 `1744051164.010143 1744050600.0 `1744051174.010136 1744050600.0 `1744051183.009127 1744050600.0 `1744051193.01014 1744050600.0 `1744051203.010154 1744050600.0 `1744051244.010135 1744050600.0 `1744051254.010137 1744050600.0 `1744051343.010136 1744050600.0 `1744051893.009131 1744051500.0 `1744052465.010151 1744052400.0 `1744052544.009127 1744052400.0 `1744052554.01014 1744052400.0 `1744052595.010135 1744052400.0 `1744052605.010133 1744052400.0 `1744052615.010135 1744052400.0 `1744052955.010134 1744052400.0 `1744052965.010138 1744052400.0 `1744052975.01014 1744052400.0 `1744053015.010136 1744052400.0 `1744053045.010138 1744052400.0 `1744053124.009141 1744052400.0 `1744053155.01015 1744052400.0 `1744053545.010149 1744053300.0 `1744053564.009126 1744053300.0 `1744053725.010147 1744053300.0 `1744053984.009126 1744053300.0 `1744054115.010141 1744053300.0 `1744054294.010141 1744054200.0 `1744054315.010137 1744054200.0 list printed: [[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], [48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82,