python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2548183 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 10831>200 please execute split_portfolio.py -i 22711925 -l 200 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['2977396'] with mtr_portfolio_ids : ['22711925'] and first list_photo_ids : [] new path : /proc/2548183/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 200 ; length of list_pids : 200 ; length of list_args : 200 time to download the photos : 31.80458092689514 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Wed May 28 18:51:00 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 972 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 2165 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 2165 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1946 wait 20 seconds inside check gpu memory 2025-05-28 18:52:31.986298: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-28 18:52:32.028084: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-28 18:52:32.029209: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-28 18:52:32.059453: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-28 18:52:32.078831: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-28 18:52:32.083435: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-28 18:52:32.116588: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-28 18:52:32.121776: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-28 18:52:32.183564: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-28 18:52:32.184910: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-28 18:52:32.186148: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-28 18:52:32.219137: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-28 18:52:32.220714: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f17b8000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-28 18:52:32.220757: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-28 18:52:32.366463: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3ab6ab80 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-28 18:52:32.366507: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-28 18:52:32.367882: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-28 18:52:32.367976: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-28 18:52:32.368005: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-28 18:52:32.368032: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-28 18:52:32.368059: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-28 18:52:32.368085: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-28 18:52:32.368110: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-28 18:52:32.368137: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-28 18:52:32.369569: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-28 18:52:32.370026: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-28 18:52:32.370963: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-28 18:52:32.370998: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-28 18:52:32.371015: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-28 18:52:32.372617: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 3137 max_wait_temp : 5 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-05-28 18:52:39.484607: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 9.010992527008057 time used to load_weights : 0.299915075302124 0it [00:00, ?it/s] 66it [00:00, 651.41it/s]temp/1748451029_2548183_1361477831_965ea018b389caf10dfa80f115007c0a_28116204.jpg temp/1748451029_2548183_1361477830_bec826848f1423bc830c52c40ff79efd_37533518.jpg temp/1748451029_2548183_1361477828_9cedd7d451f9d5c1295478afe5d99b2f_89636580.jpg temp/1748451029_2548183_1361477827_466eab221f36bc9a4cf4a132225ae11e_94117857.jpg temp/1748451029_2548183_1361477826_4b5587a4e90b8e83eb3070a2c00177f8_78775934.jpg temp/1748451029_2548183_1361477825_231ebfec78b9d15840712d0b593fe1c2_38850041.jpg temp/1748451029_2548183_1361477824_f51354827a84c4aff14dd34a6463bffe_70722500.jpg temp/1748451029_2548183_1361477823_da6cf67be8c5b43fcaa807dff5102230_96325325.jpg temp/1748451029_2548183_1361477809_b8e8323c99331581760dc4d02275266f_839388.jpg temp/1748451029_2548183_1361477808_e8a4143cd2717062493128e427a2a7d2_52483771.jpg temp/1748451029_2548183_1361477807_762614efa4d348d94cf3489b8c304cf3_75357065.jpg temp/1748451029_2548183_1361477806_80fa7226565916b425b32fa8b3b124e1_96364566.jpg temp/1748451029_2548183_1361477805_b88885fbe22ed5497c982a440bac24a4_38230132.jpg temp/1748451029_2548183_1361477804_c7d292af169f6bfbd41011710337133c_53528359.jpg temp/1748451029_2548183_1361477803_8d7f1f479b58c60ff6e937f054507d05_58093369.jpg temp/1748451029_2548183_1361477802_7dccc723cba3a1f837cf0cc9b89dd0fd_42254043.jpg temp/1748451029_2548183_1361477801_54c37dba2ff317a2ebdd5ce3a752c24d_10334932.jpg temp/1748451029_2548183_1361477800_9a31ea9f0dcc01c5161a8fea97507cb9_5223264.jpg temp/1748451029_2548183_1361477799_63bc764803869a359ccff8e57d2db18a_10603039.jpg temp/1748451029_2548183_1361477798_047525671b884a82857de0ed2f2508f9_80865393.jpg temp/1748451029_2548183_1361477792_08fb12e5ef92f36e7b661a31272d7f3a_23617080.jpg temp/1748451029_2548183_1361477790_0f41834028afdd4bdad3f129953e50fc_25446899.jpg temp/1748451029_2548183_1361477788_d1efab56794599de5a0533e769087828_6155407.jpg temp/1748451029_2548183_1361477786_1677f12d233fdbbd983a4db12a1afebf_59146905.jpg temp/1748451029_2548183_1361477784_dacb65e3d5415d1d17a272d7b4ae0f0c_73583746.jpg temp/1748451029_2548183_1361477782_dd37cc5410a45c23077ef2b270269833_60903213.jpg temp/1748451029_2548183_1361477777_610279950f6c7e69975c9b8934990aa7_77068622.jpg temp/1748451029_2548183_1361477776_3b5c7600303419ee65e4df25d8899c31_66175533.jpg temp/1748451029_2548183_1361477775_94e3bd75ba7d0c36f966a30c4645c394_68771284.jpg temp/1748451029_2548183_1361477774_f40be3575954132fb40e811b0e65529d_45014785.jpg temp/1748451029_2548183_1361477773_2ae814d2a46a83d9e5c1d756f55b9571_61630849.jpg temp/1748451029_2548183_1361477772_b950915902ce59e5cb93f28c60c36ec3_87068441.jpg temp/1748451029_2548183_1361477758_4b13358565b9c55d11ae69e26ce4cf11_75849534.jpg temp/1748451029_2548183_1361477757_d6bdc926ddbf00914a6b46cf110efc9a_34236667.jpg temp/1748451029_2548183_1361477756_0b38189e60702717726e5564438acdc1_40787034.jpg temp/1748451029_2548183_1361477755_c9f228a4b01c15a00237519301e12538_6538140.jpg temp/1748451029_2548183_1361477754_3700d94febfcd6747cd6ad65da5e5c63_82965479.jpg temp/1748451029_2548183_1361477753_7e7cafda1abf94be4e5ce0e19069a629_39688189.jpg temp/1748451029_2548183_1361477745_17ad855f44b754ae1a2ce0ba7acce754_47487635.jpg temp/1748451029_2548183_1361477744_215e4e4467a9b805092009c2019a4399_72845447.jpg temp/1748451029_2548183_1361477743_79eb26762003abe187c2df39778140d7_10352959.jpg temp/1748451029_2548183_1361477742_124a44bc8956ec7fa73469fa7d691fa3_91418936.jpg temp/1748451029_2548183_1361477741_63e5897931a3f906e674b9e447b947fc_46901563.jpg temp/1748451029_2548183_1361477740_61b0f34d12cd16c2cf155783f1857d5d_15237237.jpg temp/1748451029_2548183_1361477733_84c930d2f5687c4baf9997aa08a68ccb_77880681.jpg temp/1748451029_2548183_1361477732_661d5d74ba090ea5714c124fb37663a0_91722105.jpg temp/1748451029_2548183_1361477731_2cbec75e6138b5da49c29930f8f3905d_30324860.jpg temp/1748451029_2548183_1361477730_8b023ae96a85b179de76e5ec36633548_83842111.jpg temp/1748451029_2548183_1361477729_8b5179866957dd5d8fbebf16678e85b1_40763143.jpg temp/1748451029_2548183_1361477728_96705f0a1baf067b7291796378f74ca8_71374909.jpg temp/1748451029_2548183_1361477717_f48ce5682622722acef30fae3f1f8293_30535472.jpg temp/1748451029_2548183_1361477716_623848287a401fed13fcc7b4cae16f00_78792969.jpg temp/1748451029_2548183_1361477715_d8b885161d62d53321e611222dc715e1_41283353.jpg temp/1748451029_2548183_1361477714_2e6ade1c49fa1180f77c6732b22c1c17_60416964.jpg temp/1748451029_2548183_1361477713_fea63a7493c0ff19ebeb8f65763caf12_62550233.jpg temp/1748451029_2548183_1361477712_b3276102c11a59d78c3b04a87734511f_64850758.jpg temp/1748451029_2548183_1361477705_5fa0a2b12cb3f01683d3becf851c61ab_67662027.jpg temp/1748451029_2548183_1361477704_2bc3b7b9f25f719155775270bed2480a_65779159.jpg temp/1748451029_2548183_1361477703_6ce9985a1630e0c49dddc38bffcf3b02_32982846.jpg temp/1748451029_2548183_1361477702_80b01e4946612351a1078ae0d16ed944_21727678.jpg temp/1748451029_2548183_1361477701_47bae762f906cd299f120d6977591ffe_79623066.jpg temp/1748451029_2548183_1361477700_4953906cc9e990a173bdedcaf028f882_24903500.jpg temp/1748451029_2548183_1361477690_3c276a207485dfffd970e6795ee12088_67957567.jpg temp/1748451029_2548183_1361477689_488c5e4df2c876395a439eb1d468d7cf_77044724.jpg temp/1748451029_2548183_1361477688_113104783d6bef5b9e30662ac14bd716_69402216.jpg temp/1748451029_2548183_1361477686_78e2757c3bfb2d3da5716e62b40c4ad8_76044629.jpg temp/1748451029_2548183_1361477685_e1ef918b4c087a9fa831b4a8a5b6b7f4_45240488.jpg temp/1748451029_2548183_1361477684_12ad69b21c572a2c8563d3694fc14617_44785163.jpg temp/1748451029_2548183_1361477683_b34dea354b46b7b0c5bb7c27472335d8_32128437.jpg temp/1748451029_2548183_1361477682_2ba808220299ee4bc242a7e097b8b6e6_5253150.jpg temp/1748451029_2548183_1361477681_02432168e8a52fcac8538259ec04b675_47531895.jpg temp/1748451029_2548183_1361477680_c16e1e4ec3e271554ed54aaf1409933e_92975035.jpg temp/1748451029_2548183_1361477679_7af8d3916c2e58a0e30c11e8ba702681_30195150.jpg temp/1748451029_2548183_1361477678_ce54a9a9909116622a6d7d00c84acb27_16970492.jpg temp/1748451029_2548183_1361477664_8f9863795464b675e861a761fc8aab07_53733510.jpg temp/1748451029_2548183_1361477663_8ffef335c53ff83e06544a11c22eca6f_77520940.jpg temp/1748451029_2548183_1361477662_85ed8ee2390720aae44dc29308c73db4_59203725.jpg temp/1748451029_2548183_1361477661_d77b132bff77b1f5d681d9400a2144f5_62091933.jpg temp/1748451029_2548183_1361477659_c44ebfbda5ee8dbef9b6fa473e0f5e3a_77999276.jpg temp/1748451029_2548183_1361477658_3c679697f3ce44c649fe714eaaa3d636_42598464.jpg temp/1748451029_2548183_1361477654_fe13065c78c6a68e99c1545bc734d0ea_1762241.jpg temp/1748451029_2548183_1361477652_622cb5c18f2ed6ba39c7b4cb1d44c1e0_2789175.jpg temp/1748451029_2548183_1361477650_36248735638de05bc0d709872104855f_34273099.jpg temp/1748451029_2548183_1361477648_6cebf75d10f3120747c77b1fe80ebd7f_74674080.jpg temp/1748451029_2548183_1361477646_210c0a3c9adc0c7ce884e05404efa57b_72649073.jpg temp/1748451029_2548183_1361477644_bef1614fae956a1fb9c6b6b2bbf4046d_45891604.jpg temp/1748451029_2548183_1361477634_4524dc07a614017c0fba2bb9ad6543fa_55394274.jpg temp/1748451029_2548183_1361477633_d9f2ba111bec6f701dc66033d6548d18_73169197.jpg temp/1748451029_2548183_1361477632_e355ee8e2dcc07a3373d85bf3ae43415_34138761.jpg temp/1748451029_2548183_1361477631_bfc54f375b7acaf1785913d02ecd85d0_69574658.jpg temp/1748451029_2548183_1361477630_c0c455c8d0f920710636d6370c78cce3_66511761.jpg temp/1748451029_2548183_1361477629_5cb23dfd2f93792213537f067476dc0b_2865010.jpg temp/1748451029_2548183_1361477613_25d5456fa5ec2b0e9c8f06f2bec6d94b_80734021.jpg temp/1748451029_2548183_1361477612_c2a45946dc6267b744e8e3a69680d1e4_88138474.jpg temp/1748451029_2548183_1361477611_1aa70d43a378f977a8739ae6120eafb1_69557778.jpg temp/1748451029_2548183_1361477610_0e8c21eef79a8252db1f447e7e92605c_21609945.jpg temp/1748451029_2548183_1361477609_173742d6158b3aaaba9926bf98e5a6d9_70466216.jpg temp/1748451029_2548183_1361477608_1c241e226e885e3ed46e38d6e2287ae1_11256673.jpg temp/1748451029_2548183_1361477607_bf8c268d3067ad7136a47aab84817c97_46253084.jpg temp/1748451029_2548183_1361477606_f3bf2ce18b64c913674cc5623bfb16e2_29504223.jpg temp/1748451029_2548183_1361477604_34ca35a7f25e20f7aa061e08b715b24e_85599568.jpg 132it [00:00, 646.53it/s] 199it [00:00, 655.33it/s] 200it [00:00, 653.27it/s]2025-05-28 18:52:44.269695: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-28 18:52:44.485721: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2025-05-28 18:52:44.526256: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-28 18:52:44.539057: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-05-28 18:52:44.540976: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1748451029_2548183_1361477603_cd6bdbf36d5d39371489fd0c2c034125_3842804.jpg temp/1748451029_2548183_1361477602_6002101fef6c8716bf8ab14cb7239d4d_48199093.jpg temp/1748451029_2548183_1361477601_a5411499a542c0f8d7eb7f7608e7c432_26942807.jpg temp/1748451029_2548183_1361477590_8a420de1b87c39e0675dd70043e4de2e_31495412.jpg temp/1748451029_2548183_1361477589_1f9bd2d4946e26469cd24f5265f6d4fe_63621192.jpg temp/1748451029_2548183_1361477588_5b7b114349ae32f83eac3c4d43d3d0a9_1575405.jpg temp/1748451029_2548183_1361477587_156fabf920db0a1eb2cc93b79f801947_14143171.jpg temp/1748451029_2548183_1361477586_e55d277c4804f57163baba7d9a61f200_5012857.jpg temp/1748451029_2548183_1361477585_3a3d850aac388f600d4b92f05751d78e_60162746.jpg temp/1748451029_2548183_1361477584_436eb1d58097cf9ac3a04ec257d8fbef_14470330.jpg temp/1748451029_2548183_1361477583_58ea5289fef4882e8e8dd4d8cd13d7d6_72958139.jpg temp/1748451029_2548183_1361477582_bea8689663314a2b47d9bee07a4d5d9f_51923870.jpg temp/1748451029_2548183_1361477581_36805d04701621c677b37983780e35b2_97800925.jpg temp/1748451029_2548183_1361477580_b2ebec5fcbf6aa2054cbab5f138ac87d_88688830.jpg temp/1748451029_2548183_1361477578_d5b7ad8bebc07eedc9d9e9f2c9f2458b_47699901.jpg temp/1748451029_2548183_1361477561_4e71f4be045958c5f45a5992f46fe9c4_76007517.jpg temp/1748451029_2548183_1361477560_8f0f337cb0f9d139aa657f9d02e1befe_19091490.jpg temp/1748451029_2548183_1361477559_9121ad0d5f271cb1ae87d29f2e461aa6_69455959.jpg temp/1748451029_2548183_1361477558_7495a05647b7a47e5a5fb7cb77c82a38_26310144.jpg temp/1748451029_2548183_1361477557_41f233197f1bd7e41055b8bd87d95b8c_41922759.jpg temp/1748451029_2548183_1361477556_98b5904c52900c06e958a4a8a8d4c30d_26499419.jpg temp/1748451029_2548183_1361477550_51a94104835b93973158cf0d20924cc7_87148492.jpg temp/1748451029_2548183_1361477548_0945afad2d40d493dd88908960e042b4_16352221.jpg temp/1748451029_2548183_1361477546_c1cf6f6b5387bf86362c37138c807be5_87153507.jpg temp/1748451029_2548183_1361477545_b8d9448125d3a52874fee29ce0990929_96373526.jpg temp/1748451029_2548183_1361477544_6ad2fe7f45ac2cb999ee274c259e0231_3890856.jpg temp/1748451029_2548183_1361477543_b149c1a22202d652380ad6d6c447d29c_35841459.jpg temp/1748451029_2548183_1361477542_9feddb33185592e7acc3d321b7716071_69342598.jpg temp/1748451029_2548183_1361477541_392f53f1810ed3fe6999327b6c129ab0_57562656.jpg temp/1748451029_2548183_1361477540_6b64aa7f9c7a32d2cc455fab803baedb_17384606.jpg temp/1748451029_2548183_1361477539_62466edd92f87adf4228d30fe5ceb494_976232.jpg temp/1748451029_2548183_1361477538_9db099d5fbe0eb13b7d3fe8e4e035762_71472951.jpg temp/1748451029_2548183_1361477537_713efc109ec4c9a488da47107f32b3bf_7694554.jpg temp/1748451029_2548183_1361477527_c0adba15e89ac5a08fe19ebb9dc45725_70648387.jpg temp/1748451029_2548183_1361477526_44b01a76edf23c41f666326818cce1cc_84796002.jpg temp/1748451029_2548183_1361477525_278705cb7434fc00f403de8acb2cfcee_33285777.jpg temp/1748451029_2548183_1361477524_6256fac704b881d8c00d9ae50ff45770_94246112.jpg temp/1748451029_2548183_1361477523_eb4f49dcf9b3ce030ebab4f8e4fb149a_95481144.jpg temp/1748451029_2548183_1361477522_7e6d25364431572ef139320dc25638e4_15690149.jpg temp/1748451029_2548183_1361477515_79de9ede43daccdd7c7d07af1c515260_94303426.jpg temp/1748451029_2548183_1361477514_9f533dc41653f1e645d91934b9ac6500_41990932.jpg temp/1748451029_2548183_1361477513_50dd9146d04c5c036489237bce1b8500_88523915.jpg temp/1748451029_2548183_1361477512_d382bc25a9a0196efd6478e8360102fb_79522352.jpg temp/1748451029_2548183_1361477511_d9ff4f09bec2c0dfb2f4315d714fc9c7_96693266.jpg temp/1748451029_2548183_1361477510_672a41afb3ff19d570dee44fceaaf58c_37357197.jpg temp/1748451029_2548183_1361477497_2148413a3b2dab9dcbb28f1b67ba5adb_85974468.jpg temp/1748451029_2548183_1361477496_fccc7b2a040734475361c67bc4081e7a_49004152.jpg temp/1748451029_2548183_1361477495_28e60d9e18448f6ab90b0f7be1ebe6a2_73992348.jpg temp/1748451029_2548183_1361477494_31751eea79bf076db05fb9e28f6fbf4d_24057915.jpg temp/1748451029_2548183_1361477493_8d0179e2167871f30295e5b7fa7f974d_31117932.jpg temp/1748451029_2548183_1361477492_05e6b86d33cc45668d93d6c26341d4a7_76330015.jpg temp/1748451029_2548183_1361477479_b2430c26cc9e682d35c7f9da982c785b_88295090.jpg temp/1748451029_2548183_1361477478_d70b056c438eaa381877e26366fd2cbe_61793256.jpg temp/1748451029_2548183_1361477477_1c712109288519f317141dfd73189ca8_6897408.jpg temp/1748451029_2548183_1361477476_d928a660b5624f48bcb8b7829d754b6b_90856212.jpg temp/1748451029_2548183_1361477475_757bf91cf5e54c4ff794de1de9ea7d94_40841172.jpg temp/1748451029_2548183_1361477474_27bdee7ad2d92c52aea6492ff93c133d_66624509.jpg temp/1748451029_2548183_1361477465_6bcec864a914709895479b73db6629d0_94885726.jpg temp/1748451029_2548183_1361477463_7903fc725cca831d70cff9f9d229c29b_74902465.jpg temp/1748451029_2548183_1361477461_ee17ece29bef5f1057c58c68c4e05e08_80806529.jpg temp/1748451029_2548183_1361477459_6ff7f07deed136e5e8f8839329fe0cf4_17703695.jpg temp/1748451029_2548183_1361477457_3644de86c5e164958c6125e26dc473b5_98473597.jpg temp/1748451029_2548183_1361477456_623ac422cffb3a94e82987f8368145be_40020906.jpg temp/1748451029_2548183_1361477454_c7aab896bc4dcb9f9bf17826d9310cc8_49598225.jpg temp/1748451029_2548183_1361477453_b7f0958863c47310da12c02e3d53c39c_62284668.jpg temp/1748451029_2548183_1361477452_f723d721eb870f067d39c414886a079e_98373485.jpg temp/1748451029_2548183_1361477451_320908802454cd8d58598a9fc848ba9a_80902794.jpg temp/1748451029_2548183_1361477449_4abce51d5a0a53c4e7879106498114b3_94409145.jpg temp/1748451029_2548183_1361477448_b20c98c32daa4de4c2154fdb482ed486_18089983.jpg temp/1748451029_2548183_1361477447_a814811eba56138593d9ce4435dab0ca_55086068.jpg temp/1748451029_2548183_1361477446_480ccf8d873d466865d419a045e5ac01_20761931.jpg temp/1748451029_2548183_1361477445_d486abad80e9c92e026e97b59a0b015c_35571663.jpg temp/1748451029_2548183_1361477444_99de7611cd03b2cde3ff481f2c8f1166_84273799.jpg temp/1748451029_2548183_1361477443_a786c01d8dddf38e5df1e930a29e8b23_57579566.jpg temp/1748451029_2548183_1361477442_71519a92819f774fa56f671f61d58db7_40141583.jpg temp/1748451029_2548183_1361477428_17269ce29a1c4d407454e2c9c04beae0_82505265.jpg temp/1748451029_2548183_1361477427_b1165f8259f6e8912f4246afd39e617a_96438239.jpg temp/1748451029_2548183_1361477426_26b56c39fb90a8592fec402868ca233c_24489322.jpg temp/1748451029_2548183_1361477425_cf9e53234ec32f036803110ec9790498_18685801.jpg temp/1748451029_2548183_1361477424_60c4e94013c49783d6ac8fc7de90b897_85140229.jpg temp/1748451029_2548183_1361477423_0d2e35000341efcd43c66f9f596dd3fb_92210425.jpg temp/1748451029_2548183_1361477422_207a4e073245bbd17ea107f1fa659376_17480444.jpg temp/1748451029_2548183_1361477421_523af382d6d60157de91e78eec6ba2f5_15164264.jpg temp/1748451029_2548183_1361477420_609c84a5a795d3d5dacc3904bb853195_35786096.jpg temp/1748451029_2548183_1361477419_4933c6a9ac9fb5855d2fbf6fbb12be26_91327835.jpg temp/1748451029_2548183_1361477418_ecca97e630a410ceea5439abe9abd8fe_96875837.jpg temp/1748451029_2548183_1361477417_202bc91428bebc980c9a4d1486b8514c_87620274.jpg temp/1748451029_2548183_1361477402_e773c5eec0c902726a4800857f9f7147_97434874.jpg temp/1748451029_2548183_1361477401_88758e464643bf58601a032d29f94dd3_10690037.jpg temp/1748451029_2548183_1361477400_f7b1bab00520763a6539ce3775cdecb7_32904934.jpg temp/1748451029_2548183_1361477399_4f7caeb0e8fd1d322670b538254958a8_56474004.jpg temp/1748451029_2548183_1361477398_da91bd7b500aa5e63447ce2e48643a71_91778092.jpg temp/1748451029_2548183_1361477397_aae3c965fcd253ab6693c9d26f83c186_49109447.jpg temp/1748451029_2548183_1361477386_7eeea482a076732321b9044ccef1c0a4_74003473.jpg temp/1748451029_2548183_1361477385_971bf5cb1698afa868491a2edc153d7e_91393045.jpg temp/1748451029_2548183_1361477384_88109225d2be05bb29eea4438d38300e_3424546.jpg temp/1748451029_2548183_1361477383_5ee2afb54176fe78fd7cc8054b9be61a_41804704.jpg temp/1748451029_2548183_1361477382_431c68e2d31d3529b83180d1f9a42cb6_61483422.jpg temp/1748451029_2548183_1361477381_8121714b87ee41dd6ac8acdb93249083_45460305.jpg Found 200 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1361477831, 1361477830, 1361477828, 1361477827, 1361477826, 1361477825, 1361477824, 1361477823, 1361477809, 1361477808, 1361477807, 1361477806, 1361477805, 1361477804, 1361477803, 1361477802, 1361477801, 1361477800, 1361477799, 1361477798, 1361477792, 1361477790, 1361477788, 1361477786, 1361477784, 1361477782, 1361477777, 1361477776, 1361477775, 1361477774, 1361477773, 1361477772, 1361477758, 1361477757, 1361477756, 1361477755, 1361477754, 1361477753, 1361477745, 1361477744, 1361477743, 1361477742, 1361477741, 1361477740, 1361477733, 1361477732, 1361477731, 1361477730, 1361477729, 1361477728, 1361477717, 1361477716, 1361477715, 1361477714, 1361477713, 1361477712, 1361477705, 1361477704, 1361477703, 1361477702, 1361477701, 1361477700, 1361477690, 1361477689, 1361477688, 1361477686, 1361477685, 1361477684, 1361477683, 1361477682, 1361477681, 1361477680, 1361477679, 1361477678, 1361477664, 1361477663, 1361477662, 1361477661, 1361477659, 1361477658, 1361477654, 1361477652, 1361477650, 1361477648, 1361477646, 1361477644, 1361477634, 1361477633, 1361477632, 1361477631, 1361477630, 1361477629, 1361477613, 1361477612, 1361477611, 1361477610, 1361477609, 1361477608, 1361477607, 1361477606, 1361477604, 1361477603, 1361477602, 1361477601, 1361477590, 1361477589, 1361477588, 1361477587, 1361477586, 1361477585, 1361477584, 1361477583, 1361477582, 1361477581, 1361477580, 1361477578, 1361477561, 1361477560, 1361477559, 1361477558, 1361477557, 1361477556, 1361477550, 1361477548, 1361477546, 1361477545, 1361477544, 1361477543, 1361477542, 1361477541, 1361477540, 1361477539, 1361477538, 1361477537, 1361477527, 1361477526, 1361477525, 1361477524, 1361477523, 1361477522, 1361477515, 1361477514, 1361477513, 1361477512, 1361477511, 1361477510, 1361477497, 1361477496, 1361477495, 1361477494, 1361477493, 1361477492, 1361477479, 1361477478, 1361477477, 1361477476, 1361477475, 1361477474, 1361477465, 1361477463, 1361477461, 1361477459, 1361477457, 1361477456, 1361477454, 1361477453, 1361477452, 1361477451, 1361477449, 1361477448, 1361477447, 1361477446, 1361477445, 1361477444, 1361477443, 1361477442, 1361477428, 1361477427, 1361477426, 1361477425, 1361477424, 1361477423, 1361477422, 1361477421, 1361477420, 1361477419, 1361477418, 1361477417, 1361477402, 1361477401, 1361477400, 1361477399, 1361477398, 1361477397, 1361477386, 1361477385, 1361477384, 1361477383, 1361477382, 1361477381] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 200 time used for this insertion : 0.08756756782531738 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 103.62285828590393 time spend to save output : 0.0959782600402832 total time spend for step 0 : 103.71883654594421 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 14.57user 5.29system 2:21.53elapsed 14%CPU (0avgtext+0avgdata 1779816maxresident)k 2067696inputs+347224outputs (20003major+492767minor)pagefaults 0swaps