python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' --fifo -a 533 -l 100 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3741866 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 6696>100 please execute split_portfolio.py -i 21319675 -l 100 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2660215'] with mtr_portfolio_ids : ['21319675'] and first list_photo_ids : [] new path : /proc/3741866/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BHTTP Error 404: Not Found can't download the photo : 1343563412 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343563412_7d11a507efe29733c17a352288207552.jpg FBHTTP Error 404: Not Found can't download the photo : 1343563410 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343563410_0b7607a88b0136529fc0370d99d279c4.jpg FBHTTP Error 404: Not Found can't download the photo : 1343563407 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343563407_0ea1ca411a6783ab716a1f8c5e532446.jpg FBHTTP Error 404: Not Found can't download the photo : 1343563405 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343563405_ab0fac813084c1474cc91c0a048cdcad.jpg FBHTTP Error 404: Not Found can't download the photo : 1343563397 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343563397_a90993737deef16e0a4251ca349ec2e7.jpg FBHTTP Error 404: Not Found can't download the photo : 1343563391 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343563391_6a76ddbdd8e9e6b791afec0da0f53de6.jpg FBHTTP Error 404: Not Found can't download the photo : 1343563380 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343563380_3f451d5ed82559d7af36f1800d406b16.jpg FBHTTP Error 404: Not Found can't download the photo : 1343563356 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343563356_356df99fdea7882be84b716539c54e1a.jpg FBHTTP Error 404: Not Found can't download the photo : 1343563343 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343563343_eb94e89912ee09d10c82cc700c394344.jpg FBHTTP Error 404: Not Found can't download the photo : 1343563334 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343563334_5f2e4d0df176d357a95c43061156cd2f.jpg FBHTTP Error 404: Not Found can't download the photo : 1343563326 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343563326_1cd1536112a0c2b92c34d0c2106129c8.jpg FBHTTP Error 404: Not Found can't download the photo : 1343563322 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343563322_fa7b23ba59e2c6c14d3b860a7cbf1640.jpg FBHTTP Error 404: Not Found can't download the photo : 1343563304 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343563304_ef5bf7ee45de3211d7a84fa011c50338.jpg FBHTTP Error 404: Not Found can't download the photo : 1343563292 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343563292_c34346c3b975bd6001f1412f56aa0b75.jpg FBHTTP Error 404: Not Found can't download the photo : 1343563289 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343563289_fe7a97bbf55b6e515cd11390b97eb7e4.jpg FBHTTP Error 404: Not Found can't download the photo : 1343563285 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343563285_855dae460be1e0347ce86f464c28e374.jpg FBHTTP Error 404: Not Found can't download the photo : 1343563276 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343563276_59dc52955f5fdae090a7ecdac53436c5.jpg FBHTTP Error 404: Not Found can't download the photo : 1343563265 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343563265_45060be2d39c91b145821958422d1fff.jpg FBHTTP Error 404: Not Found can't download the photo : 1343563249 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343563249_4545df60a5c74adeac148ed42cc1aae3.jpg FBHTTP Error 404: Not Found can't download the photo : 1343563244 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343563244_d224c366028a5d78a346fa4633a91fcd.jpg FBHTTP Error 404: Not Found can't download the photo : 1343563240 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343563240_2d4c59aa2fd31df0af9b0962fe2f0149.jpg FBHTTP Error 404: Not Found can't download the photo : 1343563230 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343563230_cdc399ac416e34887dc7c28f0a476428.jpg FBHTTP Error 404: Not Found can't download the photo : 1343563081 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343563081_9d8c8e28bddfc59ea270b85bd43a63c0.jpg FBHTTP Error 404: Not Found can't download the photo : 1343561734 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343561734_6b2ca43b9946177ab9fd241328a346aa.jpg FBHTTP Error 404: Not Found can't download the photo : 1343561010 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343561010_17eaa342ff4624cef55475a9b7aa58b8.jpg FBHTTP Error 404: Not Found can't download the photo : 1343560895 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343560895_c88bef1b3944be770530b486dbb5cf66.jpg FBHTTP Error 404: Not Found can't download the photo : 1343560765 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343560765_2987b65508e962a3c2a3d045a6680910.jpg FBHTTP Error 404: Not Found can't download the photo : 1343560059 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343560059_69c10ef79b417eccabd50d04d81b5bb3.jpg FBHTTP Error 404: Not Found can't download the photo : 1343559945 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343559945_2bbecbca193a24911c5ebeef653c4ccd.jpg FBHTTP Error 404: Not Found can't download the photo : 1343559239 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343559239_cccedbfdf2894e50939a2026987b2d61.jpg FBHTTP Error 404: Not Found can't download the photo : 1343558715 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343558715_930735e9136624ae55969460ed835adb.jpg FBHTTP Error 404: Not Found can't download the photo : 1343558683 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343558683_1ebfd58ae4adf8add858981c2f414f7e.jpg FBHTTP Error 404: Not Found can't download the photo : 1343558392 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343558392_2720f405340453a72de35b868e7c0b6d.jpg FBHTTP Error 404: Not Found can't download the photo : 1343557968 complete_path : /home/admin/workarea/git/Velours/python/temp/1741804828_3741866_1343557968_d29ddeb499be1c26f9f47c7f0368cf3f.jpg Fwe have missing 34 photos in the step downloads : photo missing : [1343563412, 1343563410, 1343563407, 1343563405, 1343563397, 1343563391, 1343563380, 1343563356, 1343563343, 1343563334, 1343563326, 1343563322, 1343563304, 1343563292, 1343563289, 1343563285, 1343563276, 1343563265, 1343563249, 1343563244, 1343563240, 1343563230, 1343563081, 1343561734, 1343561010, 1343560895, 1343560765, 1343560059, 1343559945, 1343559239, 1343558715, 1343558683, 1343558392, 1343557968] try to delete the photos missing in DB Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343563412 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343563410 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343563407 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343563405 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343563397 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343563391 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343563380 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343563356 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343563343 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343563334 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343563326 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343563322 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343563304 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343563292 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343563289 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343563285 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343563276 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343563265 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343563249 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343563244 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343563240 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343563230 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343563081 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343561734 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343561010 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343560895 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343560765 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343560059 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343559945 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343559239 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343558715 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343558683 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343558392 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1343557968 length of list_filenames : 0 ; length of list_pids : 0 ; length of list_args : 0 time to download the photos : 13.012982368469238 About to test input to load empty input or other problem ret : 2 len(total_photo_id_missing) : 34 1.63user 0.60system 0:15.74elapsed 14%CPU (0avgtext+0avgdata 98764maxresident)k 0inputs+40outputs (0major+48247minor)pagefaults 0swaps