python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 1885109 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 7542>200 please execute split_portfolio.py -i 23311170 -l 200 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2946880'] with mtr_portfolio_ids : ['23311170'] and first list_photo_ids : [] new path : /proc/1885109/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 200 ; length of list_pids : 200 ; length of list_args : 200 time to download the photos : 34.50395679473877 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Mon May 26 22:31:04 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1577 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1183 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1183 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1183 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1183 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 2376 wait 20 seconds 2025-05-26 22:33:16.812088: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-26 22:33:16.852677: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-26 22:33:16.855596: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-26 22:33:16.892386: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-26 22:33:16.914853: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-26 22:33:16.920233: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-26 22:33:16.957784: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-26 22:33:16.963184: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-26 22:33:17.027730: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-26 22:33:17.028958: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-26 22:33:17.029795: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-26 22:33:17.063062: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-26 22:33:17.065132: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f5968000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-26 22:33:17.065190: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-26 22:33:17.211595: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3c2f0f70 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-26 22:33:17.211640: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-26 22:33:17.212709: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-26 22:33:17.212794: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-26 22:33:17.212829: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-26 22:33:17.212861: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-26 22:33:17.212893: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-26 22:33:17.212925: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-26 22:33:17.212957: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-26 22:33:17.212989: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-26 22:33:17.214318: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-26 22:33:17.214780: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-26 22:33:17.216100: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-26 22:33:17.216129: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-26 22:33:17.216144: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-26 22:33:17.217545: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 2376 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-05-26 22:33:24.737978: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-26 22:33:24.738667: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.72G (2921752064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-26 22:33:24.739297: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2629576704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-26 22:33:24.739879: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.20G (2366618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 9.581424474716187 time used to load_weights : 0.8670756816864014 0it [00:00, ?it/s] 71it [00:00, 707.72it/s]temp/1748291430_1885109_1360965121_193460fd0dd51423b6da1f6a888d3608.jpg temp/1748291430_1885109_1360965119_55dc9d1f42a179c90595514a5d5df1a9.jpg temp/1748291430_1885109_1360965118_27ce3eacd562d38a75c8ef7eac659e1a.jpg temp/1748291430_1885109_1360965117_52d3b1aa6f0f5b1de4a5b3a9ec862dde.jpg temp/1748291430_1885109_1360965116_fa53f8f14f38940dabb9a9a48b317078.jpg temp/1748291430_1885109_1360965115_7c5f9a3405149928b033f57bf180a0ad.jpg temp/1748291430_1885109_1360965102_9b3c79d8106d9ec2a1924f800d6601ae.jpg temp/1748291430_1885109_1360965100_ec4b1047bea9e84ae8500838d511a791.jpg temp/1748291430_1885109_1360965099_d57e1ad80520e5ded254a828473a6bee.jpg temp/1748291430_1885109_1360965098_e4268920bb3b5b6081c6cf8bd4afaa79.jpg temp/1748291430_1885109_1360965096_30452a608de84f8adad5c27e0be1c459.jpg temp/1748291430_1885109_1360965094_d62f798ca98dc2088620d2f520dead47.jpg temp/1748291430_1885109_1360965081_38e9bbdb1f080acbbde6f4212c28db66.jpg temp/1748291430_1885109_1360965078_776e49df26070593fd2caf1ba163ea98.jpg temp/1748291430_1885109_1360965076_45f3773ee849b5812dbbca61ff20c20f.jpg temp/1748291430_1885109_1360965074_cf13451d7ceac7b0b7c0b019e8700b25.jpg temp/1748291430_1885109_1360965072_9347e1526a157587f6076682a143f12d.jpg temp/1748291430_1885109_1360965069_98a563d15183d240d4ab1291013c12d0.jpg temp/1748291430_1885109_1360965056_4d437ebb052a86d2c1e0d831809ebbb8.jpg temp/1748291430_1885109_1360965053_fe7214f6929170e056ff83626e339471.jpg temp/1748291430_1885109_1360965051_9b7861296ce2bce81f95dc52ec02142a.jpg temp/1748291430_1885109_1360965050_3a28de2015550e2c9cf47548f523047b.jpg temp/1748291430_1885109_1360965049_fb56c6866553b9afc5014fbfdb3f6dc5.jpg temp/1748291430_1885109_1360965047_3e1942bc9f9531651ae24226a5f736d2.jpg temp/1748291430_1885109_1360965034_a55702343f7d59ec51595545685171ea.jpg temp/1748291430_1885109_1360965031_cdb0fcc181489ae43efb7397a673c975.jpg temp/1748291430_1885109_1360965029_e5d43b765dea7ca5a396deb922caef70.jpg temp/1748291430_1885109_1360965027_17ab8dac491a485df254c7c6fd396039.jpg temp/1748291430_1885109_1360965024_cf1dc8adc6d970f0a8c0e62e69c8cc0f.jpg temp/1748291430_1885109_1360965021_428e6f3909688a06ababd9059d24f825.jpg temp/1748291430_1885109_1360965010_dd3330744ef8d4a4d8e0e6647d3b0861.jpg temp/1748291430_1885109_1360965008_22dbb7811f6520d66f99f1668bc192a3.jpg temp/1748291430_1885109_1360965006_1d3270bb9b88252a2a0f3898dca62634.jpg temp/1748291430_1885109_1360965003_83f7923b14c6c7082a48a3f029a81720.jpg temp/1748291430_1885109_1360965001_300422ba8e060a2d6d5339a98acc0e81.jpg temp/1748291430_1885109_1360964998_b7bbbdc36c5a0e930fb1f3a56d67bda1.jpg temp/1748291430_1885109_1360964988_9899e8a2b19cfea883b27062b6175df1.jpg temp/1748291430_1885109_1360964985_5b890de7be01f8f6e649e5e24fcc362c.jpg temp/1748291430_1885109_1360964983_feea6ce3bebb5620c7be3a5f9fd65afb.jpg temp/1748291430_1885109_1360964982_c1d724355441d03d0ee5215e0e8c40aa.jpg temp/1748291430_1885109_1360964981_b99b6c2de73b29dd9c7ba3588abb9414.jpg temp/1748291430_1885109_1360964980_a1724f9330ee7f39f4d3d608f46f18b5.jpg temp/1748291430_1885109_1360964962_03282ae7d5dd6c3314ed003613f52e50.jpg temp/1748291430_1885109_1360964960_6800080f9eba4aa2370026701bf93197.jpg temp/1748291430_1885109_1360964959_0fa5d21c905a8dca4fd49f0a4a000635.jpg temp/1748291430_1885109_1360964958_35d897596925a20fc236e113e6062c2b.jpg temp/1748291430_1885109_1360964957_931f1910dab876af157ecc8dd3eabf9a.jpg temp/1748291430_1885109_1360964956_ae6822f415502c2e936db7fbf1bfc715.jpg temp/1748291430_1885109_1360964942_ea979236f89fa243ab38619ca4974d67.jpg temp/1748291430_1885109_1360964940_4325afe83784879b04ee9724145d008e.jpg temp/1748291430_1885109_1360964939_aed1e46b4602ea3fb6eba199f6d3df46.jpg temp/1748291430_1885109_1360964938_94140f5666098f182544159a674a9a4f.jpg temp/1748291430_1885109_1360964936_328a04463a347e840c82488798633ad7.jpg temp/1748291430_1885109_1360964933_1bdc46a723972a462a8293a61cd5b52c.jpg temp/1748291430_1885109_1360964923_b6c939f5bdb0220695b9762f056fa3ad.jpg temp/1748291430_1885109_1360964921_6cf48723017bfab6af0f070ec6f591d9.jpg temp/1748291430_1885109_1360964919_f88a466f1744085a0b1edf7379fbb4a9.jpg temp/1748291430_1885109_1360964916_b2bfaf9d2d2738d262b9209cc9e6ab06.jpg temp/1748291430_1885109_1360964914_f0f95f513a28550b46609c68970d754a.jpg temp/1748291430_1885109_1360964910_e1d0e95833e76e62e77edb81f4b28018.jpg temp/1748291430_1885109_1360964898_6e1ea9719e6cb34d075bc995bb1e2d88.jpg temp/1748291430_1885109_1360964897_8f09bcf842daa52d6a2b060265d5d748.jpg temp/1748291430_1885109_1360964895_0b23dd25a88615f3eaa8e9a73becaa66.jpg temp/1748291430_1885109_1360964894_4521383f2d3fe17d5de757c562eb45fa.jpg temp/1748291430_1885109_1360964893_281f7eab7f7a4e6963eb07d01a355ce7.jpg temp/1748291430_1885109_1360964891_e0c0366ba3171dae96dca2ac4976c29f.jpg temp/1748291430_1885109_1360964879_754a8a83c89ad96cb5414c10500492e6.jpg temp/1748291430_1885109_1360964877_2676eff69a59c1323221970d9a27c271.jpg temp/1748291430_1885109_1360964876_93cde9c63b9bbb25cb95134b0ec117a3.jpg temp/1748291430_1885109_1360964875_5e437c310815c0e27d5c845dd71df2d9.jpg temp/1748291430_1885109_1360964873_6cbb77b68aea09140b60a7f657beee9a.jpg temp/1748291430_1885109_1360964871_45076ad5fd8002328283275e8c92d155.jpg temp/1748291430_1885109_1360964859_1b7f4fef1dfcf733db1e47d8a98e1455.jpg temp/1748291430_1885109_1360964857_472e1ff386373f763d8dec47f3400dc1.jpg temp/1748291430_1885109_1360964855_3e44e71f3151b9e199300a7d631e744b.jpg temp/1748291430_1885109_1360964853_5c8dbc21e8fd283db233dd94a58de250.jpg temp/1748291430_1885109_1360964851_f31f4103f9af9534c45dea66420c733f.jpg temp/1748291430_1885109_1360964849_8ed4129f1492d9758c0aeb15e95d0b1b.jpg temp/1748291430_1885109_1360964838_cbe8a03055e5c6b8e3635200ef4e4565.jpg temp/1748291430_1885109_1360964835_8a40edcfdbb5c256d60b7992560c5e9f.jpg temp/1748291430_1885109_1360964832_4cd2045c9456caad271a7b148473142d.jpg temp/1748291430_1885109_1360964830_19eda1c3e88c9ddcf7dcc2e712be7480.jpg temp/1748291430_1885109_1360964828_10b4cbd54fd1ae654dc70bf424333da3.jpg temp/1748291430_1885109_1360964825_7cd567d92da7fa4521ff52383579c3d4.jpg temp/1748291430_1885109_1360964815_bf831b505d27d87b9168e9943f7254a1.jpg temp/1748291430_1885109_1360964813_c7820bd605e939d7e345f6bc0275fd67.jpg temp/1748291430_1885109_1360964811_cd482e0f5d14112021fa2262ef92bfda.jpg temp/1748291430_1885109_1360964810_f38e3894d092fec9dd27b9d5d236ca38.jpg temp/1748291430_1885109_1360964809_187a22b09be5a9d26617b7d4c4cd5d0e.jpg temp/1748291430_1885109_1360964808_3922d22ad597cc7f01175eed4dda7912.jpg temp/1748291430_1885109_1360964794_3046f683de8af5ab76670cde0cd34abd.jpg temp/1748291430_1885109_1360964793_bb27d100f6b5f3ec719b5deadc6a8723.jpg temp/1748291430_1885109_1360964792_a8ca6605782cca566d7d88d938f75efd.jpg temp/1748291430_1885109_1360964791_23acd55d4e4c6c113f13403a7015317e.jpg temp/1748291430_1885109_1360964789_6c6b16dd3591d273cb4f8744711546f4.jpg temp/1748291430_1885109_1360964787_22e67d4ae6cc4334fb20384d2bd6c3bf.jpg temp/1748291430_1885109_1360964775_49a24abaee0beeb1e82e1dfd0a79be00.jpg temp/1748291430_1885109_1360964772_f7e67532b8d5bc0b244370b23879a165.jpg temp/1748291430_1885109_1360964770_9582641b84311107bd95267efe78470c.jpg temp/1748291430_1885109_1360964768_4b788d2227bc89a8431685a88710ee6c.jpg temp/1748291430_1885109_1360964766_1fb72f99a168862e46e216547eb96be6.jpg temp/1748291430_1885109_1360964764_0979cf7e3e02d09c956639097241a6bf.jpg temp/1748291430_1885109_1360964753_3dcd0238f71700415c1f881a723f62a2.jpg temp/1748291430_1885109_1360964750_1a72c816407af9f32e2ae5ce6bf333ed.jpg temp/1748291430_1885109_1360964747_ec6b0ed3063a383db204b1e7fb2c06b7.jpg temp/1748291430_1885109_1360964745_c957bf448ddee81620a72792fcc12b41.jpg temp/1748291430_1885109_1360964742_8c551748702de975f9f3be4d8c7fbe0e.jpg temp/1748291430_1885109_1360964741_a6bb33457bc6af89fa2a8aeb88144151.jpg temp/1748291430_1885109_1360964728_83f7d5584f20153382f179fa677c34b2.jpg temp/1748291430_1885109_1360964726_e0020dc250d6cad2ef02e16528cffdc1.jpg temp/1748291430_1885109_1360964724_4200c4cfc93b74882ef0fc4bf082304f.jpg temp/1748291430_1885109_1360964722_4586e267eeba39935d060af2d9c669ad.jpg temp/1748291430_1885109_1360964720_0473059324cf5efc3f0af9147b5b139c.jpg 142it [00:00, 705.50it/s] 200it [00:00, 724.32it/s]2025-05-26 22:33:30.290590: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-26 22:33:30.606194: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2025-05-26 22:33:30.636721: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-26 22:33:30.643669: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-05-26 22:33:30.645440: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1748291430_1885109_1360964718_27d9d2fb4e7436ab01bf520782d48207.jpg temp/1748291430_1885109_1360964706_389f87f410f35eedc7ce49ecf15e26a1.jpg temp/1748291430_1885109_1360964704_0ad636dc1e0678aeb6ca772fe832a699.jpg temp/1748291430_1885109_1360964702_2687d1aa8f6122c63ac5b9194bdc8e1f.jpg temp/1748291430_1885109_1360964700_557a08fb98ffe6745be9d66a3bfb3832.jpg temp/1748291430_1885109_1360964698_43f7dbbfb1a8d27b009b3ea3ff924d6e.jpg temp/1748291430_1885109_1360964696_f63da161261d908eab223c711923fc8d.jpg temp/1748291430_1885109_1360964685_accff00f2820eebd2184d3548756f8d4.jpg temp/1748291430_1885109_1360964682_e734ed378c3aa28cb22f8097b9396525.jpg temp/1748291430_1885109_1360964680_068085bc46733a7460b6f5d92b07a75d.jpg temp/1748291430_1885109_1360964679_b00b9e1f56b02604acc6213c4c216fec.jpg temp/1748291430_1885109_1360964678_8e83a10440841a09111efdea850f5e7f.jpg temp/1748291430_1885109_1360964677_ecc8e8d23ff3074a1ac79db7e00cabff.jpg temp/1748291430_1885109_1360964662_a0a0e7f362ea79ef16fd385cac49f4fc.jpg temp/1748291430_1885109_1360964661_c5d99e3512a309f9cd959e366ee6666b.jpg temp/1748291430_1885109_1360964660_f137ebe87a167306f1d6386789ba73c0.jpg temp/1748291430_1885109_1360964659_a7c3762e59d981211cd2fbb4252dcd14.jpg temp/1748291430_1885109_1360964657_a84fd1a94f3623de6b4fd42e3821f5ab.jpg temp/1748291430_1885109_1360964655_e1e4511cdade89e33c368a4c317ebac1.jpg temp/1748291430_1885109_1360964643_984b83e6ff024c84c0a08119848450b0.jpg temp/1748291430_1885109_1360964640_580256a72ea9c27ab72382d38643d325.jpg temp/1748291430_1885109_1360964638_7b0f4d11a5ae0cb93623ee9bf4b8191a.jpg temp/1748291430_1885109_1360964636_ca880c9fd20f6930faad2d8ff9c1fbd9.jpg temp/1748291430_1885109_1360964634_beddf81d5b4ab8780a69643f6948a497.jpg temp/1748291430_1885109_1360964631_43a3085dcee57d5ee1184091100e0015.jpg temp/1748291430_1885109_1360964621_452ca3bd78b39d9bfe54777c64fad3c9.jpg temp/1748291430_1885109_1360964619_cfb9f386ad1df667f343513ee9f25e14.jpg temp/1748291430_1885109_1360964617_363bb8b1063f4ce05b5ad2bd9604ef7e.jpg temp/1748291430_1885109_1360964615_1bd10484cea388202e2af82df6739a87.jpg temp/1748291430_1885109_1360964614_1c0dcd317876c181cac89730c4a762ce.jpg temp/1748291430_1885109_1360964613_d66a3381318961e9b0fdfc5e82d4f669.jpg temp/1748291430_1885109_1360964600_1b12103186090699dd8678dc20151569.jpg temp/1748291430_1885109_1360964598_86e339ea6dfe7d08fb3c17da0fd7dc49.jpg temp/1748291430_1885109_1360964597_da70bd5c9debdd853773f20e629dfe15.jpg temp/1748291430_1885109_1360964595_df89b7c37b6e3d7617e6b8391bbbbc31.jpg temp/1748291430_1885109_1360964593_5ab669a2d30c57415a2ac4ff8ea132bc.jpg temp/1748291430_1885109_1360964591_e6b6428136e9e96a18417c7bba2847a5.jpg temp/1748291430_1885109_1360964578_0e39c7aad68599141147a91ac42baa0a.jpg temp/1748291430_1885109_1360964576_9ea29960d4555ac030a7d9fa110a6b1b.jpg temp/1748291430_1885109_1360964574_b22e544c44ed02f2e5bb0fae85efa44d.jpg temp/1748291430_1885109_1360964572_520d59d02af236af75e8cf7ba140d75f.jpg temp/1748291430_1885109_1360964570_3e36d33517f2d7fabd8397e9e67151fc.jpg temp/1748291430_1885109_1360964567_0c50d9ffd0061d834f720fd0c4c98ca8.jpg temp/1748291430_1885109_1360964535_538eb640ad39785aed2537dcc4522787.jpg temp/1748291430_1885109_1360964533_80f2198c17632b733c7fa175d524f944.jpg temp/1748291430_1885109_1360964531_04d1b61b1783f8b55c978bcefb9948dd.jpg temp/1748291430_1885109_1360964530_6cd1eeb15b7907236a7ca800a5301438.jpg temp/1748291430_1885109_1360964529_09d37f0960fa4b06f2749faacf8934f6.jpg temp/1748291430_1885109_1360964527_0ffb360b946d81ad352830820ab6f06d.jpg temp/1748291430_1885109_1360964515_717ff52fc468e835f1a6cb0ff9925717.jpg temp/1748291430_1885109_1360964514_d94eeb11a18690b4352586071a2f4ab8.jpg temp/1748291430_1885109_1360964513_7aa945a6b08c0e2182ca8b0db379873d.jpg temp/1748291430_1885109_1360964512_8ee0e59c38c7f2f093d2ddb4f9d3c8f4.jpg temp/1748291430_1885109_1360964509_fc8c875f33b9aa90f2c1c852abda165c.jpg temp/1748291430_1885109_1360964506_2966d6993ceeb58642ec7da63e9166c3.jpg temp/1748291430_1885109_1360964495_9f851347f101bc5967b4091ef1aabfd5.jpg temp/1748291430_1885109_1360964493_9d0d822a278cf81d3b06981448550c68.jpg temp/1748291430_1885109_1360964491_f2fa28f99949a8c14d4641ef323834e7.jpg temp/1748291430_1885109_1360964489_68a7fd307c00284a0a7198b71cafe1a2.jpg temp/1748291430_1885109_1360964487_ae563d942159b726423f8f10fd6506ac.jpg temp/1748291430_1885109_1360964485_2e6b5697a652290ddca3b97f968569bc.jpg temp/1748291430_1885109_1360964473_fa7dc7227f347edba058fd262d00e479.jpg temp/1748291430_1885109_1360964471_c17bd60368172ad7499e799251d5a94b.jpg temp/1748291430_1885109_1360964469_0f3d174beafe68982189c7b6181cc9db.jpg temp/1748291430_1885109_1360964468_0fd234f6e5ffdece56c7369a44e2594b.jpg temp/1748291430_1885109_1360964467_4ebb9dc64b6129acfe82e813e53c209b.jpg temp/1748291430_1885109_1360964466_afadb47be8d7154a84609afba7cd51ff.jpg temp/1748291430_1885109_1360964453_0c30d028c8c286950bfec98681e3e148.jpg temp/1748291430_1885109_1360964452_fd5b138dc917c5fa2ad4c01d7146e85b.jpg temp/1748291430_1885109_1360964451_50ee082f26bdac26196f9d97a1fdb3d0.jpg temp/1748291430_1885109_1360964450_94341ea29a9c7d8f33fc5e56d85b0726.jpg temp/1748291430_1885109_1360964448_6b6240e8cd76d70f94b8b40f76c1c607.jpg temp/1748291430_1885109_1360964446_2ad03363d208feefbea1fd67ea84755f.jpg temp/1748291430_1885109_1360964434_ffe2f377836b2a3edb9d1c559fe057bb.jpg temp/1748291430_1885109_1360964432_308d5d59d6669272f9b2b49850aff7a2.jpg temp/1748291430_1885109_1360964430_37b350a187a617f0232577de3bf56ee3.jpg temp/1748291430_1885109_1360964427_3c9190d4025ea18c4d75cbbe71897bf7.jpg temp/1748291430_1885109_1360964425_0072feef775d4f487c5a3c0b5856aaaf.jpg temp/1748291430_1885109_1360964423_9f349c1ea1da0b24a57561134ef41487.jpg temp/1748291430_1885109_1360964412_86b8c3f0fbb1edadd99f2468fba58d5c.jpg temp/1748291430_1885109_1360964410_a0d996948aa7d899fb21c53bdada6999.jpg temp/1748291430_1885109_1360964408_7a28dff135d3ada61f01070a01ca3e0d.jpg temp/1748291430_1885109_1360964407_bfb8fa620ad7530348e626fb79196d2f.jpg temp/1748291430_1885109_1360964406_8048fab2d59facd1046ee2da3b690f71.jpg temp/1748291430_1885109_1360964405_c0372aaaf086353cd9257ced781c4f8b.jpg temp/1748291430_1885109_1360964390_558b87dae4976e15579b6f7b280371e8.jpg temp/1748291430_1885109_1360964389_f74d08b18ca9aba1fef54ac682f5b295.jpg Found 200 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1360965121, 1360965119, 1360965118, 1360965117, 1360965116, 1360965115, 1360965102, 1360965100, 1360965099, 1360965098, 1360965096, 1360965094, 1360965081, 1360965078, 1360965076, 1360965074, 1360965072, 1360965069, 1360965056, 1360965053, 1360965051, 1360965050, 1360965049, 1360965047, 1360965034, 1360965031, 1360965029, 1360965027, 1360965024, 1360965021, 1360965010, 1360965008, 1360965006, 1360965003, 1360965001, 1360964998, 1360964988, 1360964985, 1360964983, 1360964982, 1360964981, 1360964980, 1360964962, 1360964960, 1360964959, 1360964958, 1360964957, 1360964956, 1360964942, 1360964940, 1360964939, 1360964938, 1360964936, 1360964933, 1360964923, 1360964921, 1360964919, 1360964916, 1360964914, 1360964910, 1360964898, 1360964897, 1360964895, 1360964894, 1360964893, 1360964891, 1360964879, 1360964877, 1360964876, 1360964875, 1360964873, 1360964871, 1360964859, 1360964857, 1360964855, 1360964853, 1360964851, 1360964849, 1360964838, 1360964835, 1360964832, 1360964830, 1360964828, 1360964825, 1360964815, 1360964813, 1360964811, 1360964810, 1360964809, 1360964808, 1360964794, 1360964793, 1360964792, 1360964791, 1360964789, 1360964787, 1360964775, 1360964772, 1360964770, 1360964768, 1360964766, 1360964764, 1360964753, 1360964750, 1360964747, 1360964745, 1360964742, 1360964741, 1360964728, 1360964726, 1360964724, 1360964722, 1360964720, 1360964718, 1360964706, 1360964704, 1360964702, 1360964700, 1360964698, 1360964696, 1360964685, 1360964682, 1360964680, 1360964679, 1360964678, 1360964677, 1360964662, 1360964661, 1360964660, 1360964659, 1360964657, 1360964655, 1360964643, 1360964640, 1360964638, 1360964636, 1360964634, 1360964631, 1360964621, 1360964619, 1360964617, 1360964615, 1360964614, 1360964613, 1360964600, 1360964598, 1360964597, 1360964595, 1360964593, 1360964591, 1360964578, 1360964576, 1360964574, 1360964572, 1360964570, 1360964567, 1360964535, 1360964533, 1360964531, 1360964530, 1360964529, 1360964527, 1360964515, 1360964514, 1360964513, 1360964512, 1360964509, 1360964506, 1360964495, 1360964493, 1360964491, 1360964489, 1360964487, 1360964485, 1360964473, 1360964471, 1360964469, 1360964468, 1360964467, 1360964466, 1360964453, 1360964452, 1360964451, 1360964450, 1360964448, 1360964446, 1360964434, 1360964432, 1360964430, 1360964427, 1360964425, 1360964423, 1360964412, 1360964410, 1360964408, 1360964407, 1360964406, 1360964405, 1360964390, 1360964389] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 200 time used for this insertion : 0.07957029342651367 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 145.80706644058228 time spend to save output : 0.0868988037109375 total time spend for step 0 : 145.8939652442932 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 15.23user 6.11system 3:06.61elapsed 11%CPU (0avgtext+0avgdata 1764728maxresident)k 2128816inputs+296144outputs (20082major+615200minor)pagefaults 0swaps