python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 4066950 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 14101>200 please execute split_portfolio.py -i 22636503 -l 200 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2977398'] with mtr_portfolio_ids : ['22636503'] and first list_photo_ids : [] new path : /proc/4066950/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 200 ; length of list_pids : 200 ; length of list_args : 200 time to download the photos : 39.484943151474 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Thu May 29 01:41:08 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 2326 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1334 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1932 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1932 wait 20 seconds inside check gpu memory 2025-05-29 01:42:39.257913: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-29 01:42:39.293162: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-29 01:42:39.295342: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-29 01:42:39.327144: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-29 01:42:39.346596: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-29 01:42:39.351182: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-29 01:42:39.385116: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-29 01:42:39.390064: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-29 01:42:39.441344: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-29 01:42:39.442599: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-29 01:42:39.443489: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-29 01:42:39.475151: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-29 01:42:39.477359: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f2ac8000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-29 01:42:39.477472: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-29 01:42:39.694728: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x43921180 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-29 01:42:39.694775: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-29 01:42:39.695724: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-29 01:42:39.695815: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-29 01:42:39.695845: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-29 01:42:39.695871: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-29 01:42:39.695897: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-29 01:42:39.695923: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-29 01:42:39.695949: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-29 01:42:39.695975: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-29 01:42:39.697321: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-29 01:42:39.697757: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-29 01:42:39.698671: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-29 01:42:39.698697: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-29 01:42:39.698712: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-29 01:42:39.700177: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 3125 max_wait_temp : 5 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-05-29 01:42:46.800385: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 8.529701709747314 time used to load_weights : 0.24480819702148438 0it [00:00, ?it/s] 60it [00:00, 591.81it/s]temp/1748475628_4066950_1361462043_eae0526688af713bf2a1f7e688a6d2bd_24579977.jpg temp/1748475628_4066950_1361462042_6f82a1ce117cd8b12427c8f98562b90d_42061998.jpg temp/1748475628_4066950_1361462041_4f175fd2cba386faac8e98bb93e8f03c_75201074.jpg temp/1748475628_4066950_1361462040_fb938fb6794fb1d2a961410eef1bbc2e_87664669.jpg temp/1748475628_4066950_1361462039_264aac98b5a0d010e4258e7b3b7d91cb_56375732.jpg temp/1748475628_4066950_1361462038_7d9bdfca58fbf6348c49ca7a9656df1d_63042507.jpg temp/1748475628_4066950_1361462036_e12b413a946b63af14d34551bab2897d_17717669.jpg temp/1748475628_4066950_1361462035_6a5c21114be9028495277240cdbebcd6_42539967.jpg temp/1748475628_4066950_1361462034_66e6038e23619dbed1aba560f4bb819a_87953815.jpg temp/1748475628_4066950_1361462033_1578cdfd12bf63eeb053c74716dc6db4_55015320.jpg temp/1748475628_4066950_1361462032_70a39b45729614beb8db9ea77af9d2a3_71176290.jpg temp/1748475628_4066950_1361462031_76d06174d93cdb0be3bb5f53fe2ff990_41740972.jpg temp/1748475628_4066950_1361462030_d496578ea0d376e797738f95491d0625_37458533.jpg temp/1748475628_4066950_1361462029_d6dd99177006d1d1378f8ddcf2cf9843_59519985.jpg temp/1748475628_4066950_1361462028_4ca6e8813a18b1c04effecf4a0d5437a_2901434.jpg temp/1748475628_4066950_1361462027_344fa4b0aabb85de8936dabe7eb35e8d_82795481.jpg temp/1748475628_4066950_1361462026_7c502fe7843d687e97b371a498d3ab2e_79851682.jpg temp/1748475628_4066950_1361462025_71d4ff60015532e42530b001ef486600_79913016.jpg temp/1748475628_4066950_1361462022_1cb6ed9fdbee30ce49c760675ae33677_80578270.jpg temp/1748475628_4066950_1361462021_65101c3990a6515f0bf3a2c76e0aa95b_66072577.jpg temp/1748475628_4066950_1361462020_c146697d8a20a6848357e5f6daef4f77_82647073.jpg temp/1748475628_4066950_1361462019_c28f4d546da60151104c0ae4f72cced7_12295214.jpg temp/1748475628_4066950_1361462018_1905205d440e280ec2604fa76e3c9808_99027588.jpg temp/1748475628_4066950_1361462017_b5805e724c618b3cb7555fd7ca761911_66966816.jpg temp/1748475628_4066950_1361462016_e1f208d9791fecd5486290595fec505f_53692128.jpg temp/1748475628_4066950_1361462015_6046c140bd9631ce478191fd29944bb7_80166237.jpg temp/1748475628_4066950_1361462014_a57b5c19f54dd454c80e41ccbe782d43_39221804.jpg temp/1748475628_4066950_1361462013_d5651a634d9df1d74f067be47428fb68_2659678.jpg temp/1748475628_4066950_1361462012_0bd292322cea1a21483c46c06c61e395_94761428.jpg temp/1748475628_4066950_1361462011_30d4ecc8075ad5f875e93848db7b1c98_41372564.jpg temp/1748475628_4066950_1361462007_49974cc392dd18e7cc851c19a93dcc28_686769.jpg temp/1748475628_4066950_1361462006_e57162da6855abb39981cc21de320b46_36861021.jpg temp/1748475628_4066950_1361462005_0992faa9abcbc133c4a2e6e4b22fc9b4_35799511.jpg temp/1748475628_4066950_1361462004_17b1adc611d81efb117652cff96bcd1d_90510473.jpg temp/1748475628_4066950_1361462003_4e07ef32d8090ceb3162611672ddc689_89368463.jpg temp/1748475628_4066950_1361462002_487c2e268b46e2a85692421ba115d4f1_85394009.jpg temp/1748475628_4066950_1361462001_e5ff6b9e2bad6d5c296dddcbcd1a11ca_74805216.jpg temp/1748475628_4066950_1361462000_cf26825ff11d042b86c4f386eccb12cd_26439640.jpg temp/1748475628_4066950_1361461999_6f78736011f664fe9cbf886c12bed7ff_72804435.jpg temp/1748475628_4066950_1361461998_5c523c36f19c4a5a0f4e3455ae591ff1_90127134.jpg temp/1748475628_4066950_1361461997_35a565a6415769668c3363612c35fb42_11839455.jpg temp/1748475628_4066950_1361461996_e289e0f9a662c3afa4689ce2b4202d6b_97371060.jpg temp/1748475628_4066950_1361461993_9bfba1223260295fa9606133559e999c_25742969.jpg temp/1748475628_4066950_1361461992_95220e6e42eaa6b01602faebacc4fd19_38936595.jpg temp/1748475628_4066950_1361461991_5e68ebd7d91a126217c9f23b51e0c9b3_61931325.jpg temp/1748475628_4066950_1361461990_b611886d4d2da66b5ff2e697dfb5cbdc_12848246.jpg temp/1748475628_4066950_1361461989_467d5751b019f94717373b395b879675_61285901.jpg temp/1748475628_4066950_1361461988_e1534f30100ae07999a770130474bf1c_58328193.jpg temp/1748475628_4066950_1361461981_7eb0e248a276f40a83b8f83365334651_87138867.jpg temp/1748475628_4066950_1361461979_5f5742f6b34b7e01eea740212065f031_30122922.jpg temp/1748475628_4066950_1361461978_393c6e115284781fa3e59065c096dacd_7559474.jpg temp/1748475628_4066950_1361461977_63416efea18831f1c918d1256ab28200_96894375.jpg temp/1748475628_4066950_1361461975_7e0bbfcc1932db83301250d01e369640_80397363.jpg temp/1748475628_4066950_1361461965_6ca04f2ce0b46e2fb68a2d530fa515d4_55707238.jpg temp/1748475628_4066950_1361461963_286d52f9ef38a03d23a90fe3165fb365_17198651.jpg temp/1748475628_4066950_1361461962_7ff606f675bf9530387f2f0b803ced29_14318329.jpg temp/1748475628_4066950_1361461961_bbb8f81abb4198240ac7212479aebbba_44470452.jpg temp/1748475628_4066950_1361461959_34a47180c9e139957f9dc6e875a816d9_46450451.jpg temp/1748475628_4066950_1361461958_7057561eab06a12f8460f7cee1535775_60942821.jpg temp/1748475628_4066950_1361461957_f6d8f94a43b6d6b46c9610afcc256cbd_69512145.jpg temp/1748475628_4066950_1361461955_5ad168a795d82efeaea22c2d3ce1cd8c_45707151.jpg temp/1748475628_4066950_1361461954_4d4a6dc6e8b65c7191fbc0aa23030f23_15166098.jpg temp/1748475628_4066950_1361461953_ff88288822dbdafbf3eb0c974aa69b19_49224248.jpg temp/1748475628_4066950_1361461951_a5ddc8ccec631f3f706d62e9b28afa4d_91929232.jpg temp/1748475628_4066950_1361461950_e1b820cad62ef5162637366d7030ad46_19105555.jpg temp/1748475628_4066950_1361461942_28bed3b1f1975a9c415656b1ca46fa13_36671942.jpg temp/1748475628_4066950_1361461941_ffbdf16ce645da9d4d2ede4aec5d8ee0_35229437.jpg temp/1748475628_4066950_1361461940_196f0b68f085588efd29f236dd2298b1_27044859.jpg temp/1748475628_4066950_1361461939_5fb5013cf5ebc6c9c8bf1a1c50e6891f_76526262.jpg temp/1748475628_4066950_1361461938_6b48cc62758856fdc43f620303d8c560_5155546.jpg temp/1748475628_4066950_1361461936_65c0f15b6f733040bf6cb85cbd864dff_56859480.jpg temp/1748475628_4066950_1361461935_d4404fe8796fc7f4a9534bf187cf472e_63648795.jpg temp/1748475628_4066950_1361461934_2cc604a0f65526fae82279774993586f_58655002.jpg temp/1748475628_4066950_1361461933_0abc6abe81856406fec5b33a255fd0b1_47587214.jpg temp/1748475628_4066950_1361461932_60fd37ff8a420b8a01ca94cfeebb71dc_56789334.jpg temp/1748475628_4066950_1361461931_fd9afe4f62ee4b5d2df04c4bfb18871a_60044520.jpg temp/1748475628_4066950_1361461930_0870701cb86dae8010e05a42f3ab128d_73284015.jpg temp/1748475628_4066950_1361461928_7ce779752b2f3c15027a66d6b9ed1ccd_23362235.jpg temp/1748475628_4066950_1361461927_43d7fddc9dd1b98e0b1156a8759dc485_2036876.jpg temp/1748475628_4066950_1361461926_3717a59dfdcd2ce0ec7b2b0f33d69607_39628684.jpg temp/1748475628_4066950_1361461924_61bb63a326ee7b1666fca5705805f0da_84260542.jpg temp/1748475628_4066950_1361461923_49e848ae7a7a9ee9d974799a7ef3a4ce_87197305.jpg temp/1748475628_4066950_1361461922_e35141b0bf4ae7b5ff0605f1e9b95dd8_88750425.jpg temp/1748475628_4066950_1361461919_b1059c93517e22d755eb96953978de5f_81424657.jpg temp/1748475628_4066950_1361461918_275bb9e64fbfd663dbe2a2a11c6dcedc_57849111.jpg temp/1748475628_4066950_1361461917_72ab42e8e624a1f9f76afcaca8ebe4b1_7616529.jpg temp/1748475628_4066950_1361461916_805db9b10d7dca462172230f416e044a_14416951.jpg temp/1748475628_4066950_1361461915_6d29c23df14d64fbbfa40a4e8e91e45c_17742845.jpg temp/1748475628_4066950_1361461913_c736f7dbd1eadfec16fcbd819bd1ad3d_92949524.jpg temp/1748475628_4066950_1361461912_210452766833bb3ddb6b0617af7120e7_75489825.jpg temp/1748475628_4066950_1361461911_07924546195fd91ada8b30a84c553c1a_27168931.jpg temp/1748475628_4066950_1361461910_78d024e08bfe373c0f40a9e106a2ad37_91654056.jpg temp/1748475628_4066950_1361461909_188f2605b27a75d1b278b74b6acaf08a_28797306.jpg temp/1748475628_4066950_1361461907_2c664a36f2b3229122e10e6b0860ee89_23392522.jpg temp/1748475628_4066950_1361461905_39a1d91afb5c309303ffa9fcf38fbe2b_4280963.jpg temp/1748475628_4066950_1361461904_5dd35a9459103f38066406431573e5a3_3067536.jpg temp/1748475628_4066950_1361461903_4d38480f99787d31a44d5791050171d3_17659514.jpg temp/1748475628_4066950_1361461902_095f47c05a55907561661f6debdb61be_89884021.jpg temp/1748475628_4066950_1361461901_797c2e08d1a3eff5e843384952f3b6b6_28719254.jpg temp/1748475628_4066950_1361461900_8d8b7c492c7eebf17bb0e787419a5bfc_73621047.jpg temp/1748475628_4066950_1361461898_747f80750bb27b774d4aa4960bf63084_4350302.jpg 129it [00:00, 644.72it/s] 194it [00:00, 638.80it/s] 200it [00:00, 637.29it/s]2025-05-29 01:42:51.249848: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-29 01:42:51.361492: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2025-05-29 01:42:51.387265: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-29 01:42:51.411080: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-05-29 01:42:51.416346: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1748475628_4066950_1361461897_4f5e4acc54513ce748a77b887edfaf8a_25326247.jpg temp/1748475628_4066950_1361461895_fd174978ee17db49daf0aa58e23dc267_55074251.jpg temp/1748475628_4066950_1361461894_c5ce5dd43bbca52540911c0c1a1074c9_38984760.jpg temp/1748475628_4066950_1361461893_677e8248db1d620d86bfec55f68fcf82_43759774.jpg temp/1748475628_4066950_1361461892_93ca0382c694d89ec5d060bae169a698_93677027.jpg temp/1748475628_4066950_1361461888_5e76826c81cec74c069d8efe67b1d84e_96360553.jpg temp/1748475628_4066950_1361461887_acbb8b7de37faef504f02ff9f73a861d_75666924.jpg temp/1748475628_4066950_1361461886_f248c1bb1ada063bfa844e90f02aae9d_61519692.jpg temp/1748475628_4066950_1361461885_ab9f3c6f691183605d2bdca20ea495ef_43830598.jpg temp/1748475628_4066950_1361461884_4e46b96d54788d19ba8d5ca102f868a1_9713640.jpg temp/1748475628_4066950_1361461883_a7ea5426bc28c6d93181c70fb249d51c_16576337.jpg temp/1748475628_4066950_1361461881_0653930b2b510dadb5375fdf48a2f57c_30175392.jpg temp/1748475628_4066950_1361461880_2fdf1c7c157b096e492b628e643ff55d_50292851.jpg temp/1748475628_4066950_1361461879_24dabada3ec084b2fe8903647df824b7_43804158.jpg temp/1748475628_4066950_1361461878_2bec38c2478bfb7007b5ca1761a3098d_76174295.jpg temp/1748475628_4066950_1361461877_6cae5adc18a81d8ad8c210a65c88a5b8_94102933.jpg temp/1748475628_4066950_1361461876_40299eec81f913af430bbdd8a472d519_59721761.jpg temp/1748475628_4066950_1361461872_5355651cf9fa50cf567ee3dafa572f37_28358804.jpg temp/1748475628_4066950_1361461871_6e0fcfc62a626e561880c47f9d94652d_70180722.jpg temp/1748475628_4066950_1361461870_763746963941bfde6717f1fd6ba36451_74989822.jpg temp/1748475628_4066950_1361461869_1a14e9250b84501e4b6b860d60ef9c93_95753226.jpg temp/1748475628_4066950_1361461868_5b8a05185496a4746cd62029a54938f0_99008306.jpg temp/1748475628_4066950_1361461867_b151474e668be55b778755941a958e86_24766676.jpg temp/1748475628_4066950_1361461863_2ff8c454b7634d78298ed94ae628d0cb_42373187.jpg temp/1748475628_4066950_1361461862_21cd61d2ea7b50198a5738676233d309_49293745.jpg temp/1748475628_4066950_1361461860_7e826b94d4696a720659b112c6b7df8c_95859307.jpg temp/1748475628_4066950_1361461859_ca7ad6e597b1443c5335f0ce4f0ad39f_14463267.jpg temp/1748475628_4066950_1361461858_001c4debc4a1c82558c6d2fe50587fbe_54098612.jpg temp/1748475628_4066950_1361461857_d179bdb24c126faee8e4a63519754871_14823787.jpg temp/1748475628_4066950_1361461855_df91a14367f0eb97a95cb45072c497a5_85256046.jpg temp/1748475628_4066950_1361461854_6d3b0ff2298458b7b2f69517d3600504_12678662.jpg temp/1748475628_4066950_1361461853_62224b723abfb4e5bce4baacd046e6da_46203319.jpg temp/1748475628_4066950_1361461852_a74f55e13ba0ceb96fb35e1cd729501b_97342064.jpg temp/1748475628_4066950_1361461851_8129a4d53defbb9662ce3babf19a2129_41477502.jpg temp/1748475628_4066950_1361461850_a7cd3a9529dc900188b03d55ac64a554_34721436.jpg temp/1748475628_4066950_1361461849_55920a7368fa355e5f298cd776a1b580_14432263.jpg temp/1748475628_4066950_1361461848_99bd9954719d22abf0f14635ba6db365_16499759.jpg temp/1748475628_4066950_1361461847_64814a2275b26a73251add50566a8e32_21375112.jpg temp/1748475628_4066950_1361461846_84869eb9053a20a6553ec8b6f960647d_83651492.jpg temp/1748475628_4066950_1361461845_30b3307baded124253fec341f539fe27_75281460.jpg temp/1748475628_4066950_1361461844_77e3edaeb1fe5905f0fe4c9ad5e38330_68547426.jpg temp/1748475628_4066950_1361461842_d19634564a0c910978fc436ca70c2a0c_40640947.jpg temp/1748475628_4066950_1361461841_728476dbff367ba3d743ce4362422f5c_12093246.jpg temp/1748475628_4066950_1361461840_f22a09f3b7b80c461499b02bfa6b65d3_19804965.jpg temp/1748475628_4066950_1361461839_65e38af93da85522f3ea59da607437a3_98860976.jpg temp/1748475628_4066950_1361461838_853840638b2802a8eee23a0fef810612_47408902.jpg temp/1748475628_4066950_1361461837_02b0dfb25ffb5f81c44f679698ed28dc_72026252.jpg temp/1748475628_4066950_1361461836_9c77a45fa2e9f1ba990130d8a557e25e_80155376.jpg temp/1748475628_4066950_1361461835_4aef604a994b2f0bf1c83b40325295c3_40533199.jpg temp/1748475628_4066950_1361461834_ea2139c5dcbdd49bae0bc484194042b7_17848634.jpg temp/1748475628_4066950_1361461833_5f7b1c7a64bc92414bc8beec7197d356_51687176.jpg temp/1748475628_4066950_1361461832_577699a8646bd9cfc571b0349f494402_90209878.jpg temp/1748475628_4066950_1361461831_53a1a7ba09ec698920445fcbbe04cc0c_58044570.jpg temp/1748475628_4066950_1361461829_2279eda6a6482616e39b7a6d99cf36b0_19593034.jpg temp/1748475628_4066950_1361461828_710b643f39775dc792f266812e884865_42565591.jpg temp/1748475628_4066950_1361461827_e82cf4447d97fad23ebd32dea0ad3d29_8347434.jpg temp/1748475628_4066950_1361461826_654a025d4dda61f8ff00714f93658139_84199300.jpg temp/1748475628_4066950_1361461825_22d3b750b0e168825693ca98203af9a3_46915972.jpg temp/1748475628_4066950_1361461824_4ba513b36af37543999e4b79edf88679_12384246.jpg temp/1748475628_4066950_1361461823_2ff6205784726c7a283466c206da7c62_79170044.jpg temp/1748475628_4066950_1361461822_4262815bfb78b99efc883afc7d13f43a_53159336.jpg temp/1748475628_4066950_1361461821_a0098cfba800326e90022626646a0077_10783167.jpg temp/1748475628_4066950_1361461820_05138642ca43de08ec480481acaf3ff1_56403299.jpg temp/1748475628_4066950_1361461819_0b95c891af23271089504c738c48e61e_9643392.jpg temp/1748475628_4066950_1361461818_03c134eaa94b3068794120033e6505d7_52617041.jpg temp/1748475628_4066950_1361461815_fee3cff1793da9257b2c0fe836d0497a_77422286.jpg temp/1748475628_4066950_1361461814_b0f8f816d83bd272914fa0343f29683f_55583648.jpg temp/1748475628_4066950_1361461813_0b67177ecf61af7e8cdacae2064b1449_50516626.jpg temp/1748475628_4066950_1361461812_c863067634b0b77615e6b88254c1207a_27862160.jpg temp/1748475628_4066950_1361461811_9d55dbf2d5abe04d99303a39c89272e9_30187938.jpg temp/1748475628_4066950_1361461810_69785426f4b496e61096213813cbb048_54075285.jpg temp/1748475628_4066950_1361461809_e9789a595b50e6306c94a9415033dac6_16717333.jpg temp/1748475628_4066950_1361461808_c6c2e58e1083c1de910976998911eb87_59956849.jpg temp/1748475628_4066950_1361461807_03764c10d0d74b1b4e5d02b416a85b78_528231.jpg temp/1748475628_4066950_1361461806_ae8808a12dff0d6cc7c1f0cb4c31b878_83802752.jpg temp/1748475628_4066950_1361461802_a632196043ad5fae4d7d2536561547bd_40277355.jpg temp/1748475628_4066950_1361461801_9a372c55b6ec25d354efc1568a649e4e_64265865.jpg temp/1748475628_4066950_1361461800_0c9ba31a99f04c1f24bec7f373b41d49_22416695.jpg temp/1748475628_4066950_1361461799_38096dab3d0a95a10fc08bf546ed00e8_66567310.jpg temp/1748475628_4066950_1361461798_2b803eeac98c50d0f8585232bc0ef28b_82171667.jpg temp/1748475628_4066950_1361461797_ae8a1f9b4ece9a8eb1350d2d82ea2826_98713482.jpg temp/1748475628_4066950_1361461789_46223a5aab6b928e8d2ff5cd9ea8f3ab_72181544.jpg temp/1748475628_4066950_1361461788_967e74cc05b9d075dbe413713c55b784_19891369.jpg temp/1748475628_4066950_1361461787_4d3d8fa2aaa9b96111f1710181e9a88c_17980051.jpg temp/1748475628_4066950_1361461786_228c0d6b653468e5e4cc852d77b6244f_76980239.jpg temp/1748475628_4066950_1361461785_52fb005bcba2e24e8e348c47e331dfe2_10055792.jpg temp/1748475628_4066950_1361461783_ddf5cdcb9dc7993fa4cb4e4d2c033084_2664499.jpg temp/1748475628_4066950_1361461782_328baea2f0f832b17f5d15442fb3996a_80840220.jpg temp/1748475628_4066950_1361461781_50f3773f0ef186c1bf73c5f71f04bcef_46494537.jpg temp/1748475628_4066950_1361461780_8b039d5c5a37dcdddd876387b8af87af_59731119.jpg temp/1748475628_4066950_1361461779_4029926b7773083fac33558a8b3dc51f_93988589.jpg temp/1748475628_4066950_1361461777_d1d20c259a9829cc66b33c9929d4daa3_87521054.jpg temp/1748475628_4066950_1361461773_8e43a875c130c3a7c1d0b22e141e6022_43164169.jpg temp/1748475628_4066950_1361461772_f8a2dd5b8f80d16dc5dd9ba57a73d354_63056317.jpg temp/1748475628_4066950_1361461771_bf77d5371b4058d9f81a84fc7535d7d8_42329093.jpg temp/1748475628_4066950_1361461770_1ee718f0dab7c7f9000cd322305f884a_93605967.jpg temp/1748475628_4066950_1361461769_91b313f51b6e493fe9af74f81a8c4b5d_94471567.jpg temp/1748475628_4066950_1361461768_5273818459b2177b398e336066a6ecdf_74612449.jpg temp/1748475628_4066950_1361461763_6abed76bc2513dea14fcfc559e87bf4a_56047799.jpg Found 200 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1361462043, 1361462042, 1361462041, 1361462040, 1361462039, 1361462038, 1361462036, 1361462035, 1361462034, 1361462033, 1361462032, 1361462031, 1361462030, 1361462029, 1361462028, 1361462027, 1361462026, 1361462025, 1361462022, 1361462021, 1361462020, 1361462019, 1361462018, 1361462017, 1361462016, 1361462015, 1361462014, 1361462013, 1361462012, 1361462011, 1361462007, 1361462006, 1361462005, 1361462004, 1361462003, 1361462002, 1361462001, 1361462000, 1361461999, 1361461998, 1361461997, 1361461996, 1361461993, 1361461992, 1361461991, 1361461990, 1361461989, 1361461988, 1361461981, 1361461979, 1361461978, 1361461977, 1361461975, 1361461965, 1361461963, 1361461962, 1361461961, 1361461959, 1361461958, 1361461957, 1361461955, 1361461954, 1361461953, 1361461951, 1361461950, 1361461942, 1361461941, 1361461940, 1361461939, 1361461938, 1361461936, 1361461935, 1361461934, 1361461933, 1361461932, 1361461931, 1361461930, 1361461928, 1361461927, 1361461926, 1361461924, 1361461923, 1361461922, 1361461919, 1361461918, 1361461917, 1361461916, 1361461915, 1361461913, 1361461912, 1361461911, 1361461910, 1361461909, 1361461907, 1361461905, 1361461904, 1361461903, 1361461902, 1361461901, 1361461900, 1361461898, 1361461897, 1361461895, 1361461894, 1361461893, 1361461892, 1361461888, 1361461887, 1361461886, 1361461885, 1361461884, 1361461883, 1361461881, 1361461880, 1361461879, 1361461878, 1361461877, 1361461876, 1361461872, 1361461871, 1361461870, 1361461869, 1361461868, 1361461867, 1361461863, 1361461862, 1361461860, 1361461859, 1361461858, 1361461857, 1361461855, 1361461854, 1361461853, 1361461852, 1361461851, 1361461850, 1361461849, 1361461848, 1361461847, 1361461846, 1361461845, 1361461844, 1361461842, 1361461841, 1361461840, 1361461839, 1361461838, 1361461837, 1361461836, 1361461835, 1361461834, 1361461833, 1361461832, 1361461831, 1361461829, 1361461828, 1361461827, 1361461826, 1361461825, 1361461824, 1361461823, 1361461822, 1361461821, 1361461820, 1361461819, 1361461818, 1361461815, 1361461814, 1361461813, 1361461812, 1361461811, 1361461810, 1361461809, 1361461808, 1361461807, 1361461806, 1361461802, 1361461801, 1361461800, 1361461799, 1361461798, 1361461797, 1361461789, 1361461788, 1361461787, 1361461786, 1361461785, 1361461783, 1361461782, 1361461781, 1361461780, 1361461779, 1361461777, 1361461773, 1361461772, 1361461771, 1361461770, 1361461769, 1361461768, 1361461763] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 200 time used for this insertion : 1.852123737335205 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 103.26311445236206 time spend to save output : 1.860917568206787 total time spend for step 0 : 105.12403202056885 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 13.71user 4.91system 2:29.20elapsed 12%CPU (0avgtext+0avgdata 1835056maxresident)k 1573552inputs+355400outputs (5423major+724222minor)pagefaults 0swaps