python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' --fifo -a 533 -l 100 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3242166 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 4876>100 please execute split_portfolio.py -i 22962323 -l 100 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2848733'] with mtr_portfolio_ids : ['22962323'] and first list_photo_ids : [] new path : /proc/3242166/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmawe have missing 100 photos in the step downloads : photo missing : [1361051904, 1361051902, 1361051900, 1361051862, 1361051861, 1361051860, 1361051814, 1361051813, 1361051812, 1361051747, 1361051746, 1361051573, 1361051566, 1361051559, 1361051553, 1361051546, 1361051540, 1361051492, 1361051489, 1361051288, 1361051287, 1361051286, 1361051243, 1361051242, 1361051239, 1361051238, 1361051204, 1361050970, 1361050969, 1361050967, 1361050966, 1361050965, 1361050963, 1361050929, 1361050927, 1361050890, 1361050889, 1361050887, 1361050885, 1361050884, 1361050883, 1361050853, 1361050852, 1361050850, 1361050849, 1361050848, 1361050802, 1361050801, 1361050721, 1361050720, 1361050719, 1361050717, 1361050716, 1361050655, 1361050654, 1361050653, 1361050652, 1361050649, 1361050648, 1361050602, 1361050601, 1361050600, 1361050598, 1361050597, 1361050594, 1361050557, 1361050556, 1361050555, 1361050554, 1361050553, 1361050551, 1361050514, 1361050464, 1361050463, 1361050462, 1361050461, 1361050458, 1361050457, 1361050419, 1361050418, 1361050416, 1361050415, 1361050413, 1361050412, 1361050365, 1361050363, 1361050362, 1361050361, 1361050360, 1361050356, 1361050304, 1361050303, 1361050302, 1361050301, 1361050299, 1361050298, 1361050248, 1361050247, 1361050244, 1361050241] try to delete the photos missing in DB Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361051904 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361051902 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361051900 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361051862 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361051861 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361051860 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361051814 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361051813 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361051812 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361051747 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361051746 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361051573 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361051566 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361051559 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361051553 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361051546 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361051540 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361051492 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361051489 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361051288 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361051287 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361051286 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361051243 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361051242 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361051239 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361051238 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361051204 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050970 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050969 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050967 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050966 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050965 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050963 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050929 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050927 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050890 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050889 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050887 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050885 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050884 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050883 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050853 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050852 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050850 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050849 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050848 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050802 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050801 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050721 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050720 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050719 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050717 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050716 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050655 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050654 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050653 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050652 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050649 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050648 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050602 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050601 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050600 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050598 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050597 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050594 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050557 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050556 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050555 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050554 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050553 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050551 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050514 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050464 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050463 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050462 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050461 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050458 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050457 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050419 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050418 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050416 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050415 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050413 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050412 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050365 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050363 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050362 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050361 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050360 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050356 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050304 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050303 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050302 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050301 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050299 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050298 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050248 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050247 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050244 Url missing : :https://www.fotonower.com/api/v1/photo/onerrorloadautomation/1361050241 length of list_filenames : 0 ; length of list_pids : 0 ; length of list_args : 0 time to download the photos : 34.29167866706848 About to test input to load empty input or other problem ret : 2 all the photo in this batch is missing ! 1.53user 1.28system 0:37.97elapsed 7%CPU (0avgtext+0avgdata 101296maxresident)k 0inputs+104outputs (9major+49477minor)pagefaults 0swaps