python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 1399370 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 18228>200 please execute split_portfolio.py -i 22740043 -l 200 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2977395'] with mtr_portfolio_ids : ['22740043'] and first list_photo_ids : [] new path : /proc/1399370/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 200 ; length of list_pids : 200 ; length of list_args : 200 time to download the photos : 40.61765098571777 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Thu May 29 09:01:11 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory 2025-05-29 09:01:13.994685: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-29 09:01:14.027538: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-29 09:01:14.027903: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-29 09:01:14.030380: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-29 09:01:14.032821: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-29 09:01:14.033183: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-29 09:01:14.035478: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-29 09:01:14.036757: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-29 09:01:14.042227: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-29 09:01:14.043456: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-29 09:01:14.043868: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-29 09:01:14.071249: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-29 09:01:14.074141: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f6894000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-29 09:01:14.074228: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-29 09:01:14.196506: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x734c6a0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-29 09:01:14.196611: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-29 09:01:14.198392: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-29 09:01:14.198674: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-29 09:01:14.198724: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-29 09:01:14.198763: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-29 09:01:14.198795: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-29 09:01:14.198823: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-29 09:01:14.198852: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-29 09:01:14.198881: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-29 09:01:14.200745: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-29 09:01:14.201063: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-29 09:01:14.202368: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-29 09:01:14.202431: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-29 09:01:14.202450: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-29 09:01:14.204333: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 3545 max_wait_temp : 1 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 9.291746854782104 time used to load_weights : 0.44642138481140137 0it [00:00, ?it/s]temp/1748502030_1399370_1361486866_e4a35f4d07f7ed123791e6c628663131_62516087.jpg temp/1748502030_1399370_1361486865_860c4c287b35655d4e21d23be358acf2_97779145.jpg temp/1748502030_1399370_1361486864_67b2b43adb8607ea0270daea2534496b_40705894.jpg temp/1748502030_1399370_1361486863_34af2b45566eaef59cf74977ccd9c8f3_46124148.jpg temp/1748502030_1399370_1361486859_e7f8e7a897271c37cfdb7d10cb06c1a5_47241652.jpg temp/1748502030_1399370_1361486858_4fef60ebe3ad1283d580a7a1d8051f90_72195278.jpg temp/1748502030_1399370_1361486857_efa7e2b641df71d479e39590e4608b6b_13670047.jpg temp/1748502030_1399370_1361486856_e24bf89f49f0dbfe91e9fcabbb89e354_53377247.jpg temp/1748502030_1399370_1361486855_7357e7088af29cdee09f5eff12d836c2_84264710.jpg temp/1748502030_1399370_1361486854_16b2e4d053b626767f6065069214605e_79707820.jpg temp/1748502030_1399370_1361486853_c78a5b35bf4021c9a20b3ed6a3c6fd1e_91039614.jpg temp/1748502030_1399370_1361486852_8621fcd42efb4ae599eb55a6202eb0eb_42114419.jpg temp/1748502030_1399370_1361486851_e9882d5c9de0c422bf80c6b2ffb408cc_36979048.jpg temp/1748502030_1399370_1361486850_5deb0ca4b374dbf91b19b2ff3c44a5a8_76450629.jpg temp/1748502030_1399370_1361486849_cf54b49d2c142c8115613ed0e51f6679_22068787.jpg temp/1748502030_1399370_1361486848_351a8eb693dfe81c99c64aecc59a699e_63558517.jpg temp/1748502030_1399370_1361486846_1237825072c455b96b0abd27c2242db0_60416832.jpg temp/1748502030_1399370_1361486845_8c392bcbc3abc3ef07d613d445445ff0_60421358.jpg temp/1748502030_1399370_1361486844_080ce075b111b25d62efa03e4df291a1_49067026.jpg temp/1748502030_1399370_1361486843_50763e1c14eaf0dd439ecdb033a10b56_99606941.jpg temp/1748502030_1399370_1361486842_2f2ed4bc7bcab3ecf6d162d07cba76eb_66389168.jpg temp/1748502030_1399370_1361486841_3af80e505fb7a2e847b7c01e7eabb32a_26736476.jpg temp/1748502030_1399370_1361486840_7d4caf6a1037fa8da054430bd3ab5bf9_45988652.jpg temp/1748502030_1399370_1361486839_c5a804aea175bd00267ebc1c3da927dc_31512937.jpg temp/1748502030_1399370_1361486838_45d0740583e5a81261ef87999184fe72_92353806.jpg temp/1748502030_1399370_1361486837_efb59b38c29e4a1b32f32805d8bc8ae8_63026775.jpg temp/1748502030_1399370_1361486836_8a034b696a8fe5cd0acf82a8a3604234_69613733.jpg temp/1748502030_1399370_1361486835_796b9e31f086b9835f1e0a4ffdc2633e_93461710.jpg temp/1748502030_1399370_1361486834_c557ceafba65b4062bc808392778c4a4_24709971.jpg temp/1748502030_1399370_1361486833_3ce115e2e6f6c5acdb6ea5517cd3acdc_16033072.jpg temp/1748502030_1399370_1361486832_7474a53caa0d77555cca568ef703ec8e_88800693.jpg temp/1748502030_1399370_1361486831_7c4d7a59e12be3e6280e2a294b0c6992_40629216.jpg temp/1748502030_1399370_1361486830_97d3821806af1ae09dbef552d78fb378_82555575.jpg temp/1748502030_1399370_1361486829_0ce65fc2e021c88c7607be94eb079d29_11804885.jpg temp/1748502030_1399370_1361486828_eac3bffebb8ff6d42f126d21c8b2b97d_33515632.jpg temp/1748502030_1399370_1361486827_3b75e59917c4e20eed3611f41ceb6539_42046824.jpg temp/1748502030_1399370_1361486826_587aeb2e99fe876084e9c21bd82b9223_9814207.jpg temp/1748502030_1399370_1361486825_013b2285cc7d51061ea1c7e055bc3e81_97145277.jpg temp/1748502030_1399370_1361486824_e9eb6eb66a2dfc98a7f16c0fe4d0ebaa_65406776.jpg temp/1748502030_1399370_1361486822_ff4b615852f183a32f67e67f9b54492e_7018756.jpg temp/1748502030_1399370_1361486821_f7f09e8866a1eace3c6c9d4b899dcd36_800193.jpg temp/1748502030_1399370_1361486820_048015a3d5834942cc54c6813778f916_65081302.jpg temp/1748502030_1399370_1361486819_e48834d012d4026b357e85809822c3c8_21766421.jpg temp/1748502030_1399370_1361486818_7af5cd0929e47b107e07143e725a4742_8311275.jpg temp/1748502030_1399370_1361486817_1ce9c4fb46398a82d04ebfbdc6bfc965_51842049.jpg temp/1748502030_1399370_1361486816_520d3d2c1d3b22d11ea9da2ae84ccbc6_93297198.jpg temp/1748502030_1399370_1361486815_7b006ef9ce8442fb2a09e1d0a02b6ae3_14949342.jpg temp/1748502030_1399370_1361486814_8efe1a37d87348eca4ab254d49a84aa1_42243399.jpg temp/1748502030_1399370_1361486813_1087534de1333310f5bf61d4779d081a_57918689.jpg temp/1748502030_1399370_1361486812_8e1994e175da3aaf6b0d4d97e3605614_41656588.jpg temp/1748502030_1399370_1361486811_a5828d54d7525f6d721c9ee8cf0935e9_52412947.jpg temp/1748502030_1399370_1361486809_8bcd34b441ac242721f1d8648341f0f5_84992627.jpg temp/1748502030_1399370_1361486808_f4445ebebe99f0f277465f0c4ef224c1_88219045.jpg temp/1748502030_1399370_1361486807_b3d585e9c732227e94e676b45357d4d8_5593613.jpg temp/1748502030_1399370_1361486806_8611bf68854736639e1e798b9b7dcf62_2058553.jpg temp/1748502030_1399370_1361486805_6edfd08e17f8f4a1225091580f69a4c9_91138099.jpg temp/1748502030_1399370_1361486804_2473dd225eb6e3037bd5c151de17949b_5083508.jpg temp/1748502030_1399370_1361486791_90cb4edc3dad8e9150f8f8e6dc1f9f04_97663692.jpg temp/1748502030_1399370_1361486790_734d40ccdaaaf724966456ce596db537_24402332.jpg temp/1748502030_1399370_1361486789_00c5b9b4540e9e354867d2eab9ecd950_6899171.jpg temp/1748502030_1399370_1361486788_55c69bc0aca7156c2304da96194b618e_78354628.jpg temp/1748502030_1399370_1361486787_08aa0d9baf3bd256b87cd08569d739f1_70275995.jpg temp/1748502030_1399370_1361486786_3a195c9b18a2460fc8dcf5897bb4ab33_25625467.jpg temp/1748502030_1399370_1361486785_c13fd6dffaa8f1fbfbb2caff8355ae2a_51402503.jpg temp/1748502030_1399370_1361486784_d08513239059a6b0590d807bd99dc1a6_32002436.jpg temp/1748502030_1399370_1361486783_3dd4116f4bcd15c9ae4b902aa4eaf9fb_43724756.jpg temp/1748502030_1399370_1361486781_026c900aa58d7e763e7f0d9ac7967424_32651556.jpg temp/1748502030_1399370_1361486780_2f1d36c6e15c1e2e352117eb656261a2_93767143.jpg temp/1748502030_1399370_1361486779_02c66dba4ff375b2d4b152ec86e41a98_63266763.jpg temp/1748502030_1399370_1361486777_58725dcc4ba39da90f19322b3e0eb81e_32762358.jpg temp/1748502030_1399370_1361486776_de8b6a0d9e22c02a7ec70885c227ee95_38272411.jpg temp/1748502030_1399370_1361486775_2f47b6b2a3cee632de3571466db27103_59042547.jpg temp/1748502030_1399370_1361486774_a8edf6be27bb81425516bb8aa747f167_61799423.jpg temp/1748502030_1399370_1361486773_637ac9ac82e0117dc27cc11bf888cd82_60025126.jpg temp/1748502030_1399370_1361486772_3db7bbe6b0f039a8aba3c276b316c3de_65515920.jpg temp/1748502030_1399370_1361486770_15dfae3b4167e371b08f5aae76b99988_22082257.jpg temp/1748502030_1399370_1361486769_106a6a171efa484697a4f5579c989597_3391635.jpg temp/1748502030_1399370_1361486768_4e87b3713a275b11f490f2e53fd96fe3_88696935.jpg temp/1748502030_1399370_1361486767_5271d51458fa9733e08b946a43007df1_11329840.jpg temp/1748502030_1399370_1361486766_fe769247b5015d066a72cb9902d55810_55900732.jpg temp/1748502030_1399370_1361486765_65fd51150db3e702a2da2332c19374c0_84178910.jpg temp/1748502030_1399370_1361486761_cf247a308f3942c8c490374920124217_45469957.jpg temp/1748502030_1399370_1361486760_2271992ed874da52240c861c4819128b_70083860.jpg temp/1748502030_1399370_1361486759_02252f11e19db698281401a1852d4191_59024958.jpg temp/1748502030_1399370_1361486758_8aa80ecaaf31153b1ed7a31e3f3703e8_10356442.jpg temp/1748502030_1399370_1361486757_8c53a4cd567289832f1e18b8b7d01c93_78625646.jpg temp/1748502030_1399370_1361486756_7da51ad90e1bf3f05004c885be22d1f1_243278.jpg temp/1748502030_1399370_1361486753_e1639a2008650503965d5197d77a8bf7_87783735.jpg temp/1748502030_1399370_1361486752_99bffa04f926bd9a82d0871c689b1c65_58902507.jpg temp/1748502030_1399370_1361486750_7deb33382c94863de9449c8218e202b7_81458325.jpg temp/1748502030_1399370_1361486749_1092596cb4e32587d9dfe6794982dfc1_78244572.jpg temp/1748502030_1399370_1361486748_002ec6d3a0c1d057d3ca519ba5d030e0_6252901.jpg temp/1748502030_1399370_1361486747_e1be3df2893dd043dac99096d842d56b_9282544.jpg temp/1748502030_1399370_1361486746_590b96abd7cce69411687adca96863b9_49224511.jpg temp/1748502030_1399370_1361486745_1b8add7315a166c3ffc845411ea63cb6_50184877.jpg temp/1748502030_1399370_1361486744_d7e52422fe5a828a1978494ff588b7e1_98159065.jpg temp/1748502030_1399370_1361486743_7664cf66a3185475c22aa1a6661ceab0_59688755.jpg temp/1748502030_1399370_1361486742_770127815954fd059430e5be5c450b8f_13152352.jpg temp/1748502030_1399370_1361486741_17df0aacb3c5840499ef4749f6de9223_35454973.jpg temp/1748502030_1399370_1361486737_7cf974310b712dd33af1e9e697a8783e_53445638.jpg temp/1748502030_1399370_1361486736_295d663b40a183e0b3d2e708004742ca_94243748.jpg 153it [00:00, 1529.19it/s] 200it [00:00, 1509.75it/s]2025-05-29 09:01:26.191444: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-29 09:01:26.482209: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-29 09:01:26.983490: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-05-29 09:01:26.988040: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1748502030_1399370_1361486734_90246d2fb5f042da53828ce9bd09f145_57836148.jpg temp/1748502030_1399370_1361486733_782505fd9dae8a2db2c63344cc23ead6_17443241.jpg temp/1748502030_1399370_1361486732_4f62993f5cab7afa91b8695d758105ae_32149131.jpg temp/1748502030_1399370_1361486731_2625f73aeb9148d7890a4e078d13c1a1_60652899.jpg temp/1748502030_1399370_1361486730_6dd55869d1c09968ec12c92a594635ce_78000481.jpg temp/1748502030_1399370_1361486729_057d6d6dbe5ecab4b0c4edb575d3f6ce_41633050.jpg temp/1748502030_1399370_1361486728_501f35a598b3aca0320800e7548e13e0_97552287.jpg temp/1748502030_1399370_1361486727_b1fc67511aecbeb33ec290fffe60df1f_70602278.jpg temp/1748502030_1399370_1361486726_7eee3e12737e44cbba3386a9f2622c83_7400364.jpg temp/1748502030_1399370_1361486725_77567c46502b9529e9725d8723cf4d3b_35306359.jpg temp/1748502030_1399370_1361486719_01083a61ced55d4782bf3a82e458173c_9068503.jpg temp/1748502030_1399370_1361486718_8fd210a2ee046fd9c47596312e04d7ed_35632884.jpg temp/1748502030_1399370_1361486717_660c22b11a6953e8e10884c834ca4b5e_73814470.jpg temp/1748502030_1399370_1361486716_5ab35593eeb81559844e177b89686f8c_81118987.jpg temp/1748502030_1399370_1361486715_7d797f279716879dc28fbef7d6de5095_94360885.jpg temp/1748502030_1399370_1361486714_46f3bbb2c63f58db850c6b6931a3097e_35541977.jpg temp/1748502030_1399370_1361486712_6953750e151d25a1396add4f165c5f1e_78073631.jpg temp/1748502030_1399370_1361486711_5814f21fddbd1a3ddbb2f6f401018e98_89447841.jpg temp/1748502030_1399370_1361486710_f31c86584d31b6b5f8ed2a31e085fc86_56902700.jpg temp/1748502030_1399370_1361486709_f85c266b79de14f37b0c49ba8dbed083_35166869.jpg temp/1748502030_1399370_1361486708_85d401f61d62b277e9f843afad6a7e09_79522790.jpg temp/1748502030_1399370_1361486707_fd9b5551b4e95c37a285734eca9e271f_50833243.jpg temp/1748502030_1399370_1361486703_337ce5093486b487def904bbd858e0dd_9956177.jpg temp/1748502030_1399370_1361486702_dab05b45e48c76b8cca60589167fcb9c_86851664.jpg temp/1748502030_1399370_1361486701_0da7aa706e226fde87cac79d74f411a1_4172477.jpg temp/1748502030_1399370_1361486700_07e2350b1a118a5bf31b638f9281708f_44937572.jpg temp/1748502030_1399370_1361486699_ebb039fb525c02b7080d5692ff07728a_28952429.jpg temp/1748502030_1399370_1361486698_2a9f7492a3d1e6df3c376ddba9f7f235_36979491.jpg temp/1748502030_1399370_1361486696_32c39e39616ea8b7bc41bd1989414162_3859650.jpg temp/1748502030_1399370_1361486695_b6cc355a5b8a024d2d45c6e5c8e2e1d4_89296030.jpg temp/1748502030_1399370_1361486694_8e3771b40f591003a611f1fc726ebe8f_71124054.jpg temp/1748502030_1399370_1361486693_74f7d1f1295cafc26ea43eae471f096e_12051154.jpg temp/1748502030_1399370_1361486692_aa713a1ec0fe5a768f02839af8503a36_10902686.jpg temp/1748502030_1399370_1361486691_74b06ab0f52b209cb8cfdea90bea8c0c_3826821.jpg temp/1748502030_1399370_1361486688_923fe24519f98b84f0386bbe66150a89_40782897.jpg temp/1748502030_1399370_1361486687_c5a487caa2bf4aeb2fbe7203900ff94a_4770607.jpg temp/1748502030_1399370_1361486686_5d846c90efa507a6faba81cdc20f6f6f_38837256.jpg temp/1748502030_1399370_1361486685_7420f5259ea863126f9f7269536d565e_34879327.jpg temp/1748502030_1399370_1361486684_4268f850edf1dc8b72498e3a8db4e023_70890688.jpg temp/1748502030_1399370_1361486683_7b0f24a35ea095bdec84b0f22715c34a_74815801.jpg temp/1748502030_1399370_1361486682_820b1e0f250cd8c813c021925687fec5_25177743.jpg temp/1748502030_1399370_1361486681_9a16731a25372a5e7dde7019a7511766_68032778.jpg temp/1748502030_1399370_1361486680_5603f17dcafbbb8573d36a388299dc2e_31975670.jpg temp/1748502030_1399370_1361486679_231af5ca99a94c4b1b14d127955dac5d_89964466.jpg temp/1748502030_1399370_1361486678_ec3df7f2066391220eaeb224c5ee6961_95461760.jpg temp/1748502030_1399370_1361486677_7bc44a85a29bfda779101c0b71f36804_58088941.jpg temp/1748502030_1399370_1361486673_1c7d10c51d79f4d087908dfa7be6f188_38655639.jpg temp/1748502030_1399370_1361486672_67c2c4e326479ff7c7a48fe36c7ebaa1_20408432.jpg temp/1748502030_1399370_1361486671_b994712baa153133e43f034896014609_30715053.jpg temp/1748502030_1399370_1361486670_9e28af910d68e7d105431423bde316ff_33767191.jpg temp/1748502030_1399370_1361486669_1b6baa627d7afc50d2b468480f4b9c2c_82096727.jpg temp/1748502030_1399370_1361486668_0e4b2b0e402d25ea7f97050b9271c761_9684868.jpg temp/1748502030_1399370_1361486667_095ef4dc68232477f25d33cc8c517b86_38345596.jpg temp/1748502030_1399370_1361486666_c6cd8702e09af90900fc14042499c342_37617315.jpg temp/1748502030_1399370_1361486665_8bdef760518ac66f19a483590026f35b_94100992.jpg temp/1748502030_1399370_1361486664_8cae2c34827e7db44eac313767bfcb13_6255323.jpg temp/1748502030_1399370_1361486663_ca4f0c32a957c8605787e7a5612686f9_3118877.jpg temp/1748502030_1399370_1361486662_c214b3330c3fa82a2133f3690e5e49d9_92717858.jpg temp/1748502030_1399370_1361486661_d63454031f8c1bfa7a41740cd20a409a_71330978.jpg temp/1748502030_1399370_1361486660_79207cca097044cc1229feae7f7bb726_17875957.jpg temp/1748502030_1399370_1361486659_ec06ef7259704b751071cbe62f81b043_85073543.jpg temp/1748502030_1399370_1361486658_b2335112dc0b3d004a98b98df640dee9_97132333.jpg temp/1748502030_1399370_1361486657_15bfa4184b2b6d36abf3b7c091065647_26106424.jpg temp/1748502030_1399370_1361486656_43cf87a64ad83eba197962ec30075712_9172585.jpg temp/1748502030_1399370_1361486655_e178afe6c4af719fdcd8cb4dea282317_9315584.jpg temp/1748502030_1399370_1361486654_b6c7585258ac63f43bed60ce16b3c75d_13381432.jpg temp/1748502030_1399370_1361486653_241738aaa730ce3a920985ac1a61e976_84761895.jpg temp/1748502030_1399370_1361486652_452b451dd50b0ccc02fd958043f8a505_66835413.jpg temp/1748502030_1399370_1361486651_d8411249f7eeb1768b96b17d51a7423e_78923819.jpg temp/1748502030_1399370_1361486650_e59ced71997ea9955e6fca20fb592314_49847242.jpg temp/1748502030_1399370_1361486646_71741bdbf3f625d5f5c981698f2db6df_24391428.jpg temp/1748502030_1399370_1361486645_bb3420fe43cf621b5ac4ba2778227881_1277534.jpg temp/1748502030_1399370_1361486644_3a2619727971623254c8d7207bd5d785_5651117.jpg temp/1748502030_1399370_1361486643_cc8e58b084993f2d9fb5b6eeb5ee99a1_7789501.jpg temp/1748502030_1399370_1361486642_d2e56c845115295eb5ea9d3bdd0feea3_3169543.jpg temp/1748502030_1399370_1361486641_6fc8728810e8251e904926692f8186d7_2379967.jpg temp/1748502030_1399370_1361486640_115d6ccf13dab06cba5dd07caafe92a9_91061349.jpg temp/1748502030_1399370_1361486639_ac9ac9836693ab29827a2e8ce43b518f_43333435.jpg temp/1748502030_1399370_1361486638_a2d63a89a4461f82f8916a6355fcdd54_81933277.jpg temp/1748502030_1399370_1361486637_a106e4611147fed1d6bdf304db7256ae_21571981.jpg temp/1748502030_1399370_1361486636_9bd8ce4ccc7b15ed3370767075a6905f_68027149.jpg temp/1748502030_1399370_1361486635_3d3f87403e7fa3102b76d42339055083_37722654.jpg temp/1748502030_1399370_1361486631_4f6375f42677a24868583a8db55da934_93506352.jpg temp/1748502030_1399370_1361486630_4370beef6db628ddd36004803d3d6c2d_63500450.jpg temp/1748502030_1399370_1361486629_5974b697bdbf368cd3187d5f8adcfdd3_23370142.jpg temp/1748502030_1399370_1361486628_831f19f18cd3600c82c13fd5a672c374_36014431.jpg temp/1748502030_1399370_1361486627_1f127aee2cffe3f1e4b95329b48c2887_87258437.jpg temp/1748502030_1399370_1361486626_aa7fc025f3987ee2e1200f25db130330_74999239.jpg temp/1748502030_1399370_1361486625_46d5a69559a91ae83d6785fbf16c942d_1297069.jpg temp/1748502030_1399370_1361486624_482999cf3ea00e75b133148ce39c899b_11111561.jpg temp/1748502030_1399370_1361486623_d276ba051e0436a7a5d7a0c1aa23c85f_40106088.jpg temp/1748502030_1399370_1361486622_36a955dc02930a4e79a9306d96dd71f5_3206871.jpg temp/1748502030_1399370_1361486621_242de4c4e9b065a72cce601a440b934b_14976129.jpg temp/1748502030_1399370_1361486620_2e8be131aab92497167aae915f0a72f0_70672223.jpg temp/1748502030_1399370_1361486617_47341b55c001f7412a0802ff58043fa5_5350850.jpg temp/1748502030_1399370_1361486616_e57540a9f8b221eb523b07d1017dd7fc_28663749.jpg temp/1748502030_1399370_1361486615_4a89c32121e1789f3bacd6ce44cf7f0a_8278046.jpg temp/1748502030_1399370_1361486614_5ac68895339a9d9d042679ad3777b155_81938635.jpg temp/1748502030_1399370_1361486613_3aa64c9f027375ce12c01b70bc7c0e70_80841564.jpg Found 200 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1361486866, 1361486865, 1361486864, 1361486863, 1361486859, 1361486858, 1361486857, 1361486856, 1361486855, 1361486854, 1361486853, 1361486852, 1361486851, 1361486850, 1361486849, 1361486848, 1361486846, 1361486845, 1361486844, 1361486843, 1361486842, 1361486841, 1361486840, 1361486839, 1361486838, 1361486837, 1361486836, 1361486835, 1361486834, 1361486833, 1361486832, 1361486831, 1361486830, 1361486829, 1361486828, 1361486827, 1361486826, 1361486825, 1361486824, 1361486822, 1361486821, 1361486820, 1361486819, 1361486818, 1361486817, 1361486816, 1361486815, 1361486814, 1361486813, 1361486812, 1361486811, 1361486809, 1361486808, 1361486807, 1361486806, 1361486805, 1361486804, 1361486791, 1361486790, 1361486789, 1361486788, 1361486787, 1361486786, 1361486785, 1361486784, 1361486783, 1361486781, 1361486780, 1361486779, 1361486777, 1361486776, 1361486775, 1361486774, 1361486773, 1361486772, 1361486770, 1361486769, 1361486768, 1361486767, 1361486766, 1361486765, 1361486761, 1361486760, 1361486759, 1361486758, 1361486757, 1361486756, 1361486753, 1361486752, 1361486750, 1361486749, 1361486748, 1361486747, 1361486746, 1361486745, 1361486744, 1361486743, 1361486742, 1361486741, 1361486737, 1361486736, 1361486734, 1361486733, 1361486732, 1361486731, 1361486730, 1361486729, 1361486728, 1361486727, 1361486726, 1361486725, 1361486719, 1361486718, 1361486717, 1361486716, 1361486715, 1361486714, 1361486712, 1361486711, 1361486710, 1361486709, 1361486708, 1361486707, 1361486703, 1361486702, 1361486701, 1361486700, 1361486699, 1361486698, 1361486696, 1361486695, 1361486694, 1361486693, 1361486692, 1361486691, 1361486688, 1361486687, 1361486686, 1361486685, 1361486684, 1361486683, 1361486682, 1361486681, 1361486680, 1361486679, 1361486678, 1361486677, 1361486673, 1361486672, 1361486671, 1361486670, 1361486669, 1361486668, 1361486667, 1361486666, 1361486665, 1361486664, 1361486663, 1361486662, 1361486661, 1361486660, 1361486659, 1361486658, 1361486657, 1361486656, 1361486655, 1361486654, 1361486653, 1361486652, 1361486651, 1361486650, 1361486646, 1361486645, 1361486644, 1361486643, 1361486642, 1361486641, 1361486640, 1361486639, 1361486638, 1361486637, 1361486636, 1361486635, 1361486631, 1361486630, 1361486629, 1361486628, 1361486627, 1361486626, 1361486625, 1361486624, 1361486623, 1361486622, 1361486621, 1361486620, 1361486617, 1361486616, 1361486615, 1361486614, 1361486613] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 200 time used for this insertion : 0.15723681449890137 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 15.880937099456787 time spend to save output : 0.16822385787963867 total time spend for step 0 : 16.049160957336426 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 16.28user 4.59system 1:02.85elapsed 33%CPU (0avgtext+0avgdata 2101032maxresident)k 27816inputs+329528outputs (187major+659852minor)pagefaults 0swaps