python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3131815 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 237>200 please execute split_portfolio.py -i 31292033 -l 200 size over we load limit photo not treated list_input_json: {} Date 2026-10-10 16:30:31.567973 Current got : datou_id : 533, datou_cur_ids : ['4398658'] with mtr_portfolio_ids : ['31292033'] and first list_photo_ids : [] new path : /proc/3131815/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 200 ; length of list_pids : 200 ; length of list_args : 200 time to download the photos : 68.18215274810791 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Sat Oct 10 16:31:39 2026 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory 2026-10-10 16:31:42.949441: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2026-10-10 16:31:42.981167: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2026-10-10 16:31:42.981436: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2026-10-10 16:31:42.983650: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2026-10-10 16:31:42.985603: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2026-10-10 16:31:42.985951: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2026-10-10 16:31:42.987954: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2026-10-10 16:31:42.988938: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2026-10-10 16:31:42.992745: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2026-10-10 16:31:42.993796: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2026-10-10 16:31:42.994159: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2026-10-10 16:31:43.017364: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3492960000 Hz 2026-10-10 16:31:43.019107: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f1b2c000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2026-10-10 16:31:43.019134: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2026-10-10 16:31:43.224283: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x229c8860 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2026-10-10 16:31:43.224324: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2026-10-10 16:31:43.225221: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2026-10-10 16:31:43.225296: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2026-10-10 16:31:43.225324: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2026-10-10 16:31:43.225350: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2026-10-10 16:31:43.225376: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2026-10-10 16:31:43.225401: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2026-10-10 16:31:43.225427: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2026-10-10 16:31:43.225453: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2026-10-10 16:31:43.226833: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2026-10-10 16:31:43.226906: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2026-10-10 16:31:43.227790: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2026-10-10 16:31:43.227814: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2026-10-10 16:31:43.227828: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2026-10-10 16:31:43.229351: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 3470 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:14 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:15 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:15 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:16 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:16 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:16 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:17 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:17 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2026-03-04 13:32:01 create time in s3 : 2026-03-04 12:24:27 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2026-03-04 13:32:01 create time in s3 : 2026-03-04 12:24:28 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2026-03-04 13:32:02 create time in s3 : 2026-03-04 12:24:31 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2026-03-04 13:32:02 create time in s3 : 2026-03-04 12:24:31 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2026-03-04 13:32:04 create time in s3 : 2026-03-04 12:24:36 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 7.500944375991821 time used to load_weights : 0.49860692024230957 0it [00:00, ?it/s] 7it [00:00, 60.14it/s] 14it [00:00, 60.41it/s] 21it [00:00, 60.15it/s] 28it [00:00, 60.26it/s] 38it [00:00, 72.76it/s] 50it [00:00, 87.62it/s] 63it [00:00, 98.36it/s] 75it [00:00, 104.85it/s] 88it [00:00, 109.44it/s] 101it [00:01, 112.73it/s]temp/1791642631_3131815_1420257870_66c2913eb053ca966f49320c0823596c.jpg temp/1791642631_3131815_1420257868_9908074ef1699d640ce65bb68c3121ed.jpg temp/1791642631_3131815_1420257867_8d8e6b9cf8bd0197ca853e9bec270cd9.jpg temp/1791642631_3131815_1420257866_61322b813d93fe2472a00c4ef905570a.jpg temp/1791642631_3131815_1420257865_a08b355a568139585612d5758d0ae89f.jpg temp/1791642631_3131815_1420257864_0feec799639185fde6d73357d2124f71.jpg temp/1791642631_3131815_1420257860_ec6442873318b377a6511324bdd575be.jpg temp/1791642631_3131815_1420257859_3c10bff31b690369802592bd71391117.jpg temp/1791642631_3131815_1420257858_2d0f7ef83809cbc2841278891f966e77.jpg temp/1791642631_3131815_1420257857_be26166e32e4d28a94a522b5b117f9e6.jpg temp/1791642631_3131815_1420256010_f25f690796c3e4eefc5e6ee11f7da434.jpg temp/1791642631_3131815_1420256009_e06085cb29159299febbe324cf180d3b.jpg temp/1791642631_3131815_1420256008_36e7765e784e10f9efa05a31d4b725a9.jpg temp/1791642631_3131815_1420256006_5916d614dec4900dd8315a6c536b9317.jpg temp/1791642631_3131815_1420256005_4d3c6906c5eeea3353f68a50ac77fab2.jpg temp/1791642631_3131815_1420256003_74804b42e7baf4b55b75a6f73ffb09be.jpg temp/1791642631_3131815_1420255990_254aeda607b83778054003415d651ceb.jpg temp/1791642631_3131815_1420255989_a752ea3dde05dec4e0a83077e58ba88d.jpg temp/1791642631_3131815_1420255988_cc97a3da1c094213b65f935056d4bc42.jpg temp/1791642631_3131815_1420255987_fc841c825165951fc923c5448e29acb7.jpg temp/1791642631_3131815_1420255986_f833aa82d0969c6f34ce41924380ee35.jpg temp/1791642631_3131815_1420255985_1a16b6184d588c5bfbeb7a94fcb8e2bc.jpg temp/1791642631_3131815_1420255495_ac61c3032da6c8143630186c34bdd354.jpg temp/1791642631_3131815_1420255477_7d13fd3589d0b320502c3c7864ebc150.jpg temp/1791642631_3131815_1420255476_4bb5f883b220fede70187d1e2784660f.jpg temp/1791642631_3131815_1420255475_9d755f4ed97c7286ebe550594afb38b0.jpg temp/1791642631_3131815_1420255473_c7ce98799d857adb91c26d458da90a47.jpg temp/1791642631_3131815_1420255471_50a012fa7f4f902ba351de60d37ecfad.jpg temp/1791642631_3131815_1420254353_d11e096a74cfde5d4c2b6654acfd21e7.jpg temp/1791642631_3131815_1420254302_f1fc8ad1c5d947bf7d2db56d1d189d1b.jpg temp/1791642631_3131815_1420254249_b1025ebc33f5460e2b90e46796601d0f.jpg temp/1791642631_3131815_1420254219_950ebd10c01b7814b81a9b29735c1408.jpg temp/1791642631_3131815_1420254218_7d4a25a502a11d2f672e259b0b28180a.jpg temp/1791642631_3131815_1420254216_de2466f456505ea7df3c5ad295d6878e.jpg temp/1791642631_3131815_1420252209_7238e9270b5a64e5bcacd71cc16a46a7.jpg temp/1791642631_3131815_1420252206_061b0c34bb7ba00ee0f5df500211f774.jpg temp/1791642631_3131815_1420252205_115e1e3b4c4060202c2af8ed38479323.jpg temp/1791642631_3131815_1420252204_bbcd12eb492800ed6baa274509dedddd.jpg temp/1791642631_3131815_1420252203_34341cbb171507f856e18d133d6a02ac.jpg temp/1791642631_3131815_1420252202_553a575f359bbff17165db16f1e15cc9.jpg temp/1791642631_3131815_1420251744_59ef8f845a094a0cc89d70b47d65ba02.jpg temp/1791642631_3131815_1420251684_5d3fa7b126ef3bff10558fdb02554b3e.jpg temp/1791642631_3131815_1420251632_3a722f1cec876709036fe5ceb03a421c.jpg temp/1791642631_3131815_1420251578_eb2e7211f639ab5a445bdc26cb6af025.jpg temp/1791642631_3131815_1420251529_706c215750a39f7f250e05408a5437ba.jpg temp/1791642631_3131815_1420251468_27b4c79948271e7c280b7567d641d0f9.jpg temp/1791642631_3131815_1420251381_5d00b61de2bb6a1e2ed17466dec94cc2.jpg temp/1791642631_3131815_1420251375_1ca62616771345290459c1d1d6acaff1.jpg temp/1791642631_3131815_1420251356_86f49fe6fa34f4aae8efb108e668812d.jpg temp/1791642631_3131815_1420251325_815f84d804bbae59ca1c5d09556b502f.jpg temp/1791642631_3131815_1420251252_ccc212f5137407cd29bade63a581d17e.jpg temp/1791642631_3131815_1420251251_4d6d543b35e9f8948d4ba9d884a82875.jpg temp/1791642631_3131815_1420251008_0fdadf9c2920265be233b49c158b3cd1.jpg temp/1791642631_3131815_1420251007_18d1c7aa887487a30c5551d4a72289be.jpg temp/1791642631_3131815_1420251006_78b3de4d7306b6d0169071a9e625c520.jpg temp/1791642631_3131815_1420251005_85ac4e2d44a8dd6454f475d621209818.jpg temp/1791642631_3131815_1420251004_a2c608d789f866b8d941b5d235c21be5.jpg temp/1791642631_3131815_1420251003_6eb7b60de6c91376cabfaa78ba9994e5.jpg temp/1791642631_3131815_1420250991_8a48d3a1699fac50b3ad872e2f1da0bf.jpg temp/1791642631_3131815_1420250990_c72a376ee13347358a9b1855ac572a00.jpg temp/1791642631_3131815_1420250989_81733d5490bef266c7b4846f7b58625c.jpg temp/1791642631_3131815_1420250988_2421008b906061136f98a0ca7c5a8ab7.jpg temp/1791642631_3131815_1420250986_c3a5520df896993dcfeb89790ca01d57.jpg temp/1791642631_3131815_1420250984_f1d923b86d42c582c9e0c885c82841f2.jpg temp/1791642631_3131815_1420250935_2718499b031a5f2f85508423b58b60b9.jpg temp/1791642631_3131815_1420250882_ec2072da20efb664b0b67db51101555e.jpg temp/1791642631_3131815_1420250856_8484c7121a1fc4fea828af44ae1bc71c.jpg temp/1791642631_3131815_1420250854_b37ef99f32c323f3d429643536d1a081.jpg temp/1791642631_3131815_1420250853_f0e92b74afcd54da3d0de7fe2a53fbaa.jpg temp/1791642631_3131815_1420250823_37f004b95b598a617980c6891fbdc8d6.jpg temp/1791642631_3131815_1420249961_534b534a60401cd2f268f39e9620c0ab.jpg temp/1791642631_3131815_1420249770_ee726a1beb8e0f96bbe80b28a151a30c.jpg temp/1791642631_3131815_1420249720_0a011c25bf087aa18abeb555359ab9a2.jpg temp/1791642631_3131815_1420249660_20d93dba9a33e8ba616446fa29dadc90.jpg temp/1791642631_3131815_1420249612_e64a25e4a866c71373daf103c859eaa6.jpg temp/1791642631_3131815_1420249534_2c4207e1c29110c0c42bfb9d912cc41c.jpg temp/1791642631_3131815_1420247624_b12a41573ae8512a84f6bd902a0f4141.jpg temp/1791642631_3131815_1420247623_c521accedfde7da4d97c6d50218447c5.jpg temp/1791642631_3131815_1420247622_5df6a3b6bc5d7fdb951749d75db1799b.jpg temp/1791642631_3131815_1420247621_fe64aea9798825057ace280cc15967e0.jpg temp/1791642631_3131815_1420247620_cf64eb8f13c6943f936b3c3ca50df453.jpg temp/1791642631_3131815_1420247619_b036dabc6dda6ba1e9cec62b4a4573ff.jpg temp/1791642631_3131815_1420246939_030c79d028e1f32cdca8edc58e1f0977.jpg temp/1791642631_3131815_1420246937_a5b3e408ff5a4e4fc79bef0fb119f177.jpg temp/1791642631_3131815_1420246935_903c50bb4f71ac2760f3d906ce99e49e.jpg temp/1791642631_3131815_1420246934_f85e7479a63ce9a225f730c4ced2772c.jpg temp/1791642631_3131815_1420246933_59f36d1cf205cc0e3f6a53200e8be0ad.jpg temp/1791642631_3131815_1420246932_8dec5b0c1bf8b59e49d79835ef410018.jpg temp/1791642631_3131815_1420246923_f162fff255a67a3af29e1b2c195521df.jpg temp/1791642631_3131815_1420246919_4f79c578348efd08e9d0fa7db12345ed.jpg temp/1791642631_3131815_1420246918_4b1991d09a899a1418fc250326a4ca87.jpg temp/1791642631_3131815_1420246917_7f4863d0df73fdb93e4aafcc7e2e5fd5.jpg temp/1791642631_3131815_1420246916_30a1c69324e063aa85d4dcbc7f632b6a.jpg temp/1791642631_3131815_1420246915_bd5f09d12072bede8dfca302eb08c11c.jpg temp/1791642631_3131815_1420244969_c10700c5f8c40ec10c9e7de4d2db077b.jpg temp/1791642631_3131815_1420244910_6de4374d03bb75af7ff783f60df91a1b.jpg temp/1791642631_3131815_1420244880_5ab09fc276524c94a81d20e7d0b978ee.jpg temp/1791642631_3131815_1420244879_68f242174c57fb3641f9a6d412ebe00d.jpg temp/1791642631_3131815_1420244878_68e26b98fc20b1e9ee9f9511c6993f68.jpg temp/1791642631_3131815_1420244877_39961489e285335358f23071acbd564e.jpg temp/1791642631_3131815_1420244476_eaa0d1cf9d8536f97d6707a709bf79fe.jpg temp/1791642631_3131815_1420244475_0c4a625bcb49adba3adc35d92bd09675.jpg temp/1791642631_3131815_1420244474_a9cb2807dfcf259b4d1c0b854c028b1c.jpg temp/1791642631_3131815_1420244473_894bbd4982eb95c8c50840f77f1b8063.jpg temp/1791642631_3131815_1420244472_c1312a8a745367e69728a69051f71bff.jpg temp/1791642631_3131815_1420244471_42819fdcba88e8cc2e9c7441254fa420.jpg temp/1791642631_3131815_1420241844_6f4df4c7553223ce8149f69f2ad67740.jpg temp/1791642631_3131815_1420241842_108600579c9009886f69bdc2d0e29055.jpg temp/1791642631_3131815_1420241841_f9b9ddfc2046b061d2feb1074c552f03.jpg temp/1791642631_3131815_1420241840_d27281fbdf0dbc9d663289a85d617c68.jpg temp/1791642631_3131815_1420241839_936fdba27fd6d74383f63d0a54a06cad.jpg temp/1791642631_3131815_1420241838_aa87595ac2b8fec1585d70a5010f7d1a.jpg temp/1791642631_3131815_1420241828_0d60607afd3dc23e57ed4dae3b98b411.jpg 114it [00:01, 115.04it/s] 126it [00:01, 115.66it/s] 138it [00:01, 115.85it/s] 150it [00:01, 115.72it/s] 162it [00:01, 116.03it/s] 174it [00:01, 110.09it/s] 178it [00:01, 99.75it/s] temp/1791642631_3131815_1420241826_fdfaec6c2e7846145647b8ef359696fe.jpg temp/1791642631_3131815_1420241825_2a7f83703adbbf118552fdd3b749f4f8.jpg temp/1791642631_3131815_1420241822_198cfccc540fe42a8b293753af94b1bd.jpg temp/1791642631_3131815_1420241797_a049db1655be424e94e145172d6b33a6.jpg temp/1791642631_3131815_1420241778_644f0f4a7b66819d3abd8aa0610159a5.jpg temp/1791642631_3131815_1420241488_5277e06c8a5c65b6b76fa4d3ffbff83d.jpg temp/1791642631_3131815_1420241487_86d259766d47374321997166b1effccf.jpg temp/1791642631_3131815_1420241486_66d342eab68320d8abfeb731e0bec0c1.jpg temp/1791642631_3131815_1420241485_ea816b042fc21a59db93cb227e4c7d89.jpg temp/1791642631_3131815_1420240599_ce3b6bd0cda7c483f125b95878484855.jpg temp/1791642631_3131815_1420240544_182166197ef60c103f66e4d15156b415.jpg temp/1791642631_3131815_1420240492_17f469a4ece48dc7cb6c030227d9aebc.jpg temp/1791642631_3131815_1420240438_fd15bb21a112280c81a1f52313623925.jpg temp/1791642631_3131815_1420240385_8386536f09de17cb4281055557281a41.jpg temp/1791642631_3131815_1420240329_49f20d5bcafc84b3f4ffbb7a5ff266f0.jpg temp/1791642631_3131815_1420237965_863bef3d461ef48bfa48b0f96c346319.jpg temp/1791642631_3131815_1420237933_a3b759e9fcd9b8339909e71ea3ac0428.jpg temp/1791642631_3131815_1420237861_ecdfdc9ac921502e95cd7f44820db3b4.jpg temp/1791642631_3131815_1420237776_54f620d4b8c254733b29b0fba34a821a.jpg temp/1791642631_3131815_1420237671_1c906abd1beab83ea56352a1925079df.jpg temp/1791642631_3131815_1420237561_3a777cb41bdc5f0f3cc0166139fe4e2d.jpg temp/1791642631_3131815_1420236105_d5404b1927431979545d36e5062e80a7.jpg temp/1791642631_3131815_1420236055_b378fd114d40581f5259a38aff4f06f9.jpg temp/1791642631_3131815_1420236004_975367641a167be5b856b7b05323a0c9.jpg temp/1791642631_3131815_1420235948_7a53dfbb0e1c0346b22538fde3a13159.jpg temp/1791642631_3131815_1420235894_bce446a59f9e6a10330b88494d7aac37.jpg temp/1791642631_3131815_1420235843_a27f3e56dc8b1e70ec3ddea606ebce10.jpg temp/1791642631_3131815_1420234347_755d67702a6945570e750397f13771ac.jpg temp/1791642631_3131815_1420234346_974bc8bab57bf25a054b59c7a3c3a409.jpg temp/1791642631_3131815_1420234344_484590f7b4e1ff7172130ee47a0f8c8e.jpg temp/1791642631_3131815_1420234342_44c26de202e1c14e406dc1d2f28c4049.jpg temp/1791642631_3131815_1420234341_4a139e6b3d39cd9beb4c70fb13336c24.jpg temp/1791642631_3131815_1420234340_ead90cd431c4a0f85ccec468e3c33e17.jpg temp/1791642631_3131815_1420234329_b683529d15e6d1134739cda19bba3739.jpg temp/1791642631_3131815_1420234327_459500513460cd6566550b1c17ca3e76.jpg temp/1791642631_3131815_1420234326_e54c0dbcc10ae243d5d7cae8cdb077c7.jpg temp/1791642631_3131815_1420234325_e3c8c89867fb670312de792c0bea1a64.jpg temp/1791642631_3131815_1420234324_d2dff8233d0d6c0476c40d1d33daa1d5.jpg temp/1791642631_3131815_1420234323_0fd11be7c8cc19892448cdce748a0c69.jpg temp/1791642631_3131815_1420234313_aaa8fcfcdec4235d303885789b30e14c.jpg temp/1791642631_3131815_1420234310_1f158ce3497886e2c2d29fa019b8cd1a.jpg temp/1791642631_3131815_1420234309_5cc86d851819849974ffb75999a33d4b.jpg temp/1791642631_3131815_1420234308_4950ac57a4f566a1020c26a963dc3ece.jpg temp/1791642631_3131815_1420234307_eefbffd8b0b28994da974e0e83b54b03.jpg temp/1791642631_3131815_1420234306_2d119bb83e1d165be8c442fe91c2bb4e.jpg temp/1791642631_3131815_1420234304_dc15d850b620bde9d3576d527ad57859.jpg temp/1791642631_3131815_1420234303_5869357c5049db4ca039bd9322864705.jpg temp/1791642631_3131815_1420234302_4c19d28a58f0e87acd7cb589e4ef2c09.jpg temp/1791642631_3131815_1420234301_ea7b84d02c726ddf7577e015be331d89.jpg temp/1791642631_3131815_1420234284_0f0ff2240ea47ff06496f409ace2ffdb.jpg temp/1791642631_3131815_1420234283_fd29a3f4e344253d2624d03756696515.jpg temp/1791642631_3131815_1420234282_ee4805c2851295a9f6ad49d1dfa15cf2.jpg temp/1791642631_3131815_1420234281_0c32a591bfac3bd57adc29a6d1964891.jpg temp/1791642631_3131815_1420234280_357664e45a5932081e842eeea1d88a31.jpg temp/1791642631_3131815_1420234279_e27af5191037ee60fe6f1f54f44dad8c.jpg temp/1791642631_3131815_1420233895_73ccd2c4285ed0bc158b28e178996366.jpg temp/1791642631_3131815_1420233894_95df2b6c8e4bc4f10fdd2a7190e8ff12.jpg temp/1791642631_3131815_1420233893_7c8ec58b6005c6e6eb4487f1953207dd.jpg temp/1791642631_3131815_1420233892_baf25bc0c2a2091183720670697dda4d.jpg temp/1791642631_3131815_1420233891_9cd86028afdc478c2d9799dc2f96f391.jpg temp/1791642631_3131815_1420233890_ce3606042cca862234224f659cd7b6dc.jpg temp/1791642631_3131815_1420232473_d640966e74862392c728d80d6135df0f.jpg temp/1791642631_3131815_1420232416_6fffa30e252088e543bbb75248ca088e.jpg temp/1791642631_3131815_1420232339_c18d6b3ea1a57c82d49281b35871ca91.jpg temp/1791642631_3131815_1420232324_69bda63ccebefbbfea5ed3268854b630.jpg temp/1791642631_3131815_1420232322_e2ae709cef2e5bb07a77881b4ad21637.jpg ERROR in datou_step_exec, will save and exit ! [Errno 28] No space left on device: 'temp/1791642631_3131815_1420232322_e2ae709cef2e5bb07a77881b4ad21637.jpg' -> 'tempdir_3131815/temp/image000000178_1791642631_3131815_1420232322_e2ae709cef2e5bb07a77881b4ad21637.jpg' File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2340, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2534, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3171, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 257, in predict_image_paths shutil.copyfile(this_path, os.path.join("tempdir_" + process_id, "temp", File "/usr/lib/python3.8/shutil.py", line 275, in copyfile _fastcopy_sendfile(fsrc, fdst) File "/usr/lib/python3.8/shutil.py", line 166, in _fastcopy_sendfile raise err from None File "/usr/lib/python3.8/shutil.py", line 152, in _fastcopy_sendfile sent = os.sendfile(outfd, infd, offset, blocksize) [1420257870, 1420257868, 1420257867, 1420257866, 1420257865, 1420257864, 1420257860, 1420257859, 1420257858, 1420257857, 1420256010, 1420256009, 1420256008, 1420256006, 1420256005, 1420256003, 1420255990, 1420255989, 1420255988, 1420255987, 1420255986, 1420255985, 1420255495, 1420255477, 1420255476, 1420255475, 1420255473, 1420255471, 1420254353, 1420254302, 1420254249, 1420254219, 1420254218, 1420254216, 1420252209, 1420252206, 1420252205, 1420252204, 1420252203, 1420252202, 1420251744, 1420251684, 1420251632, 1420251578, 1420251529, 1420251468, 1420251381, 1420251375, 1420251356, 1420251325, 1420251252, 1420251251, 1420251008, 1420251007, 1420251006, 1420251005, 1420251004, 1420251003, 1420250991, 1420250990, 1420250989, 1420250988, 1420250986, 1420250984, 1420250935, 1420250882, 1420250856, 1420250854, 1420250853, 1420250823, 1420249961, 1420249770, 1420249720, 1420249660, 1420249612, 1420249534, 1420247624, 1420247623, 1420247622, 1420247621, 1420247620, 1420247619, 1420246939, 1420246937, 1420246935, 1420246934, 1420246933, 1420246932, 1420246923, 1420246919, 1420246918, 1420246917, 1420246916, 1420246915, 1420244969, 1420244910, 1420244880, 1420244879, 1420244878, 1420244877, 1420244476, 1420244475, 1420244474, 1420244473, 1420244472, 1420244471, 1420241844, 1420241842, 1420241841, 1420241840, 1420241839, 1420241838, 1420241828, 1420241826, 1420241825, 1420241822, 1420241797, 1420241778, 1420241488, 1420241487, 1420241486, 1420241485, 1420240599, 1420240544, 1420240492, 1420240438, 1420240385, 1420240329, 1420237965, 1420237933, 1420237861, 1420237776, 1420237671, 1420237561, 1420236105, 1420236055, 1420236004, 1420235948, 1420235894, 1420235843, 1420234347, 1420234346, 1420234344, 1420234342, 1420234341, 1420234340, 1420234329, 1420234327, 1420234326, 1420234325, 1420234324, 1420234323, 1420234313, 1420234310, 1420234309, 1420234308, 1420234307, 1420234306, 1420234304, 1420234303, 1420234302, 1420234301, 1420234284, 1420234283, 1420234282, 1420234281, 1420234280, 1420234279, 1420233895, 1420233894, 1420233893, 1420233892, 1420233891, 1420233890, 1420232473, 1420232416, 1420232339, 1420232324, 1420232322, 1420232304, 1420231428, 1420231427, 1420231426, 1420231413, 1420231412, 1420231376, 1420230115, 1420230113, 1420230112, 1420230111, 1420230110, 1420230109, 1420229751, 1420229750, 1420229749, 1420229748, 1420229747, 1420229746, 1420229743, 1420229742] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 200 time used for this insertion : 0.15390443801879883 save_final ERROR in last step tfhub_classification2, [Errno 28] No space left on device: 'temp/1791642631_3131815_1420232322_e2ae709cef2e5bb07a77881b4ad21637.jpg' -> 'tempdir_3131815/temp/image000000178_1791642631_3131815_1420232322_e2ae709cef2e5bb07a77881b4ad21637.jpg' time spend for datou_step_exec : 14.526092767715454 time spend to save output : 0.1620655059814453 total time spend for step 0 : 14.6881582736969 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 14.20user 9.71system 1:30.31elapsed 26%CPU (0avgtext+0avgdata 1766748maxresident)k 739288inputs+5687296outputs (263major+638628minor)pagefaults 0swaps