python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 12 -a ' -a 533 -l 200 --rand=1 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3235124 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 249>200 please execute split_portfolio.py -i 31146218 -l 200 size over we load limit photo not treated list_input_json: {} Date 2026-06-11 11:20:29.475016 Current got : datou_id : 533, datou_cur_ids : ['4385230'] with mtr_portfolio_ids : ['31146218'] and first list_photo_ids : [] new path : /proc/3235124/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 200 ; length of list_pids : 200 ; length of list_args : 200 time to download the photos : 50.90307545661926 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Thu Jun 11 11:21:21 2026 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1531 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1137 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1137 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1137 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 2330 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 2330 wait 20 seconds 2026-06-11 11:23:30.131431: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2026-06-11 11:23:30.163414: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2026-06-11 11:23:30.164773: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2026-06-11 11:23:30.170096: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2026-06-11 11:23:30.182656: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2026-06-11 11:23:30.184213: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2026-06-11 11:23:30.208359: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2026-06-11 11:23:30.212317: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2026-06-11 11:23:30.243810: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2026-06-11 11:23:30.245012: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2026-06-11 11:23:30.246219: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2026-06-11 11:23:30.278597: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493010000 Hz 2026-06-11 11:23:30.280970: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f5034000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2026-06-11 11:23:30.280998: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2026-06-11 11:23:30.492053: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2f374ee0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2026-06-11 11:23:30.492095: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2026-06-11 11:23:30.492785: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2026-06-11 11:23:30.492867: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2026-06-11 11:23:30.492895: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2026-06-11 11:23:30.492922: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2026-06-11 11:23:30.492947: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2026-06-11 11:23:30.492972: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2026-06-11 11:23:30.492998: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2026-06-11 11:23:30.493023: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2026-06-11 11:23:30.494079: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2026-06-11 11:23:30.494553: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2026-06-11 11:23:30.495229: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2026-06-11 11:23:30.495247: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2026-06-11 11:23:30.495259: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2026-06-11 11:23:30.496346: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 2330 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2026-06-11 11:23:36.971984: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-06-11 11:23:36.972739: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.72G (2921752064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-06-11 11:23:36.973468: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2629576704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-06-11 11:23:36.974234: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.20G (2366618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:14 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:15 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:15 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:16 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:16 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:16 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:17 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2026-03-04 13:31:58 create time in s3 : 2026-03-04 12:24:17 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2026-03-04 13:32:01 create time in s3 : 2026-03-04 12:24:27 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2026-03-04 13:32:01 create time in s3 : 2026-03-04 12:24:28 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2026-03-04 13:32:02 create time in s3 : 2026-03-04 12:24:31 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2026-03-04 13:32:02 create time in s3 : 2026-03-04 12:24:31 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2026-03-04 13:32:04 create time in s3 : 2026-03-04 12:24:36 model_weights.h5 already exist and didn't need to update desc size : 2048 Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= module (KerasLayer) (None, 2048) 23561152 _________________________________________________________________ Rungis_amount_dechets_fall_2 (None, 5) 10245 ================================================================= Total params: 23,571,397 Trainable params: 10,245 Non-trainable params: 23,561,152 _________________________________________________________________ Loading Weights... time used to create the model : 8.000533103942871 time used to load_weights : 0.4541778564453125 0it [00:00, ?it/s] 7it [00:00, 61.11it/s] 14it [00:00, 63.85it/s] 21it [00:00, 64.05it/s] 28it [00:00, 64.19it/s] 35it [00:00, 64.61it/s] 42it [00:00, 63.87it/s] 49it [00:00, 62.95it/s] 56it [00:00, 60.75it/s] 63it [00:01, 61.32it/s] 70it [00:01, 61.60it/s] 77it [00:01, 59.98it/s] 84it [00:01, 58.38it/s] 90it [00:01, 55.56it/s] 96it [00:01, 53.05it/s] 102it [00:01, 50.54it/s] 109it [00:01, 55.05it/s]temp/1781169630_3235124_1418283852_f9e4b89db9823c1e92603d65c2db3df5.jpg temp/1781169630_3235124_1418283851_29b0904098e18f1157cf616cf88b405d.jpg temp/1781169630_3235124_1418283792_99d27028353d2fa2af7c8097d513a78c.jpg temp/1781169630_3235124_1418283791_76420caabc5d857708a4a03b7b4640d1.jpg temp/1781169630_3235124_1418283790_53be367449533b46f4624bcbf1cde009.jpg temp/1781169630_3235124_1418283785_e74feec1215ec1b6d7c16c1c2e4db802.jpg temp/1781169630_3235124_1418283784_2d5027cce3c7cf43dabb32f19ce22c67.jpg temp/1781169630_3235124_1418283782_91d1ff1f0f1796648fad0a129ff54964.jpg temp/1781169630_3235124_1418282386_0a694cae9d65734e6c6a801539c8aace.jpg temp/1781169630_3235124_1418282385_ac391790ff282333119ca7af58a03591.jpg temp/1781169630_3235124_1418282384_356def1d330ad4e6346404403fe409d4.jpg temp/1781169630_3235124_1418282382_c4e79f70c76a42467534e98ccc4adce4.jpg temp/1781169630_3235124_1418282379_bc5c28338c1e503299902f361a48666c.jpg temp/1781169630_3235124_1418282378_9f3c320995925da14282e1bb37fadd92.jpg temp/1781169630_3235124_1418282333_0ae8b7935a98ca8262bb2d01f20e6ab5.jpg temp/1781169630_3235124_1418282332_b6d080676539ee0e60a0b4a9c31f7a2b.jpg temp/1781169630_3235124_1418282331_5b4219396e79ba05e4f9821fe7964b86.jpg temp/1781169630_3235124_1418282330_c87b96757a6e97aa9458d0c844f47cd7.jpg temp/1781169630_3235124_1418282329_164ce4b6858e522822f6fe08cd76d072.jpg temp/1781169630_3235124_1418282328_4c92af66968110553c62564408a2da89.jpg temp/1781169630_3235124_1418282250_1b68fb0cad469c7b9a7e1358c0fc7cab.jpg temp/1781169630_3235124_1418282248_c4376af044a09edd0e8154126fc4b624.jpg temp/1781169630_3235124_1418282246_21718588b4e96ef2c9c12a8470981d9e.jpg temp/1781169630_3235124_1418282244_fbd51fba9bed09b32de4b1d2d181df0e.jpg temp/1781169630_3235124_1418282241_3fe1e42de01a3420a56c0ee7d0f50d4d.jpg temp/1781169630_3235124_1418282239_796343e4d3024fa6584e6fd43bfedebe.jpg temp/1781169630_3235124_1418282161_56189f3f208a825c4bffccc89256443e.jpg temp/1781169630_3235124_1418282160_7f64b3e23989104bf5f6c8c42162fdfb.jpg temp/1781169630_3235124_1418282159_8df61ea997016a228ebaa6bfd907847c.jpg temp/1781169630_3235124_1418282158_84710a56b8027ca5562bc89b7ba93bb5.jpg temp/1781169630_3235124_1418282157_6d0efea11b34a691873cb52b9cae3d79.jpg temp/1781169630_3235124_1418282156_9932d9ea005375c0302d5dc89ca707c4.jpg temp/1781169630_3235124_1418282111_d0c32aa4da4186ce186b69a2e4484b62.jpg temp/1781169630_3235124_1418282110_8a93c6ba06b64c94a4ccc8cab900b5d3.jpg temp/1781169630_3235124_1418282108_6b8c38f20470a8a383953cd377a93a5a.jpg temp/1781169630_3235124_1418282106_2d2ff91a9e2f9c61be7b177bcdd3b809.jpg temp/1781169630_3235124_1418282105_215e1249fa2a7fb945efda0b3a686d09.jpg temp/1781169630_3235124_1418282103_463bd290a675f259cfdce1f452f490ca.jpg temp/1781169630_3235124_1418281767_c11492de9f62915fcfc3dc55d4fa6fe5.jpg temp/1781169630_3235124_1418281766_e63be5a379660fc98ced5e2a1b65fca0.jpg temp/1781169630_3235124_1418281765_85568bf871d492014603157e97ee01f7.jpg temp/1781169630_3235124_1418281763_03f4e9b465d90127eeede1b2e56488b2.jpg temp/1781169630_3235124_1418281761_8b290974d96bbe2f1ea55b8e02652023.jpg temp/1781169630_3235124_1418281759_2e156660583c3af7201d2e6e9381720f.jpg temp/1781169630_3235124_1418281676_9d5247f25704c9ae51752c3e07f3ab4d.jpg temp/1781169630_3235124_1418281674_4db316131ddad0ca4a4d6a667fb0839d.jpg temp/1781169630_3235124_1418281672_df1b7b02867f9e3f67066e565554ff12.jpg temp/1781169630_3235124_1418281670_e42caa3423152d88bf45f58716a5ed01.jpg temp/1781169630_3235124_1418281668_1f86408cac72a879d8893897af8fb10c.jpg temp/1781169630_3235124_1418281667_a32030a624e9c74ae599c9a3c0a8b8b6.jpg temp/1781169630_3235124_1418281623_86b4219c7e372c358a9d9223c922acde.jpg temp/1781169630_3235124_1418281622_4e76cd2953b4c582011c1c77772328ba.jpg temp/1781169630_3235124_1418281621_eafa9baa87cbff9424c83b81abd01cc9.jpg temp/1781169630_3235124_1418281620_58bee09007d3c49097733862bda568ac.jpg temp/1781169630_3235124_1418281619_f9d92f19c8946dbc77ec3b5d86070892.jpg temp/1781169630_3235124_1418281617_1e6d0a712e18a647704eb9f7913e2761.jpg temp/1781169630_3235124_1418281575_2f07882230d7fdd3df5ce65bc7fea2be.jpg temp/1781169630_3235124_1418281573_2aa9626b6f01b2a9253404d4ba94a2b0.jpg temp/1781169630_3235124_1418281570_213b5aa51e044ada59e14a209f264edb.jpg temp/1781169630_3235124_1418281568_7531e00cfbb651da63301de8c1663c56.jpg temp/1781169630_3235124_1418281566_af5ecf27f0b4be339c2f4d91c59ecef3.jpg temp/1781169630_3235124_1418281565_021dc65492be8dad79fe13452dec4929.jpg temp/1781169630_3235124_1418279482_6314f1a1c69810f0dd9615efb7fb9bd1.jpg temp/1781169630_3235124_1418279480_ef0938627c5a32e7f1bed437887e8b6c.jpg temp/1781169630_3235124_1418279470_07e77d8434ef39c5e783f8aaf7647bad.jpg temp/1781169630_3235124_1418279468_4228aa92d152cf6570e4edbebdedde02.jpg temp/1781169630_3235124_1418279466_4a5a27f4701db5a1931aa5ca20910d19.jpg temp/1781169630_3235124_1418279465_a5be8aa599d5f8bddce3a6ab87aaed94.jpg temp/1781169630_3235124_1418279426_c2072b48d558c520bca3de01632e4dfc.jpg temp/1781169630_3235124_1418279424_8ae39a18f2137a5c1bdc7e0a3efccbcc.jpg temp/1781169630_3235124_1418279422_161fc6e2bb277faa65c685787bd3e699.jpg temp/1781169630_3235124_1418279421_633ee63f87f070fcbd18d532602ed372.jpg temp/1781169630_3235124_1418279419_971faf07d94518cbb110b02c8bc0f487.jpg temp/1781169630_3235124_1418279416_a0671474ddd70fa0d6176f56e01b643c.jpg temp/1781169630_3235124_1418278622_09c267f1c92dd672f97abd9b416cbdb3.jpg temp/1781169630_3235124_1418278621_8f1511a1c3dde3dcc5e6e4342674d742.jpg temp/1781169630_3235124_1418278620_6bfdd5fa188ce316a2697980dff21ba1.jpg temp/1781169630_3235124_1418278619_27e3686e5a33074d0f19c0a37d934ef4.jpg temp/1781169630_3235124_1418278617_884367c4cc3328d9cb3979e6a8d2d226.jpg temp/1781169630_3235124_1418278615_790780784b525fae24d1ae0285fb93aa.jpg temp/1781169630_3235124_1418277058_bf992eaf92835aba93718185bb459be3.jpg temp/1781169630_3235124_1418277056_cbae3b423308e3d8691d8b9288648e04.jpg temp/1781169630_3235124_1418277055_10ccd6150f1b670d70f40d12dfbd37a8.jpg temp/1781169630_3235124_1418277054_daa8e2f4cd65ccc95f76ba000c4c1673.jpg temp/1781169630_3235124_1418277053_240d12d604fc5eec39a5b6838246b624.jpg temp/1781169630_3235124_1418277052_6f53e414f5c1adc39b2e57042c0b0242.jpg temp/1781169630_3235124_1418277013_0e4045d0c6f47dd27b4012633b22fbad.jpg temp/1781169630_3235124_1418277012_13e46123e4fe3e644f4858780859b98a.jpg temp/1781169630_3235124_1418277011_5a74b87091964e3f9268a967ba187d54.jpg temp/1781169630_3235124_1418277010_ad06f3d7740cfc3165db42d64282c30e.jpg temp/1781169630_3235124_1418277009_354c381a0664e293261fbcf675d2553e.jpg temp/1781169630_3235124_1418277008_84fa78d87e9c766073396e3d4a864194.jpg temp/1781169630_3235124_1418276963_a57529582db48816be89b388cca14739.jpg temp/1781169630_3235124_1418276961_dc1f8ddc8d5c8819e96abc774dd11938.jpg temp/1781169630_3235124_1418276959_86cc4c1931fd35646555a6a72818481d.jpg temp/1781169630_3235124_1418276957_e3638e1dd52b2486dcb12f4d5c1df0c4.jpg temp/1781169630_3235124_1418276955_417959f92dcf6b801d23bb9ef673d6c9.jpg temp/1781169630_3235124_1418276952_fd382899364875f24e3afb52e0acfd24.jpg temp/1781169630_3235124_1418276902_9655fbd723630f4004a4c6ee81c3eb5e.jpg temp/1781169630_3235124_1418276900_3917185c0c5d816d168042c5b8dfc413.jpg temp/1781169630_3235124_1418276896_45154588cf79a752c143920939d00ab4.jpg temp/1781169630_3235124_1418276894_49f338ad7e37894e35fe67d5696a6010.jpg temp/1781169630_3235124_1418276892_cc3dedb21a5c97cd4b6935b10d64351b.jpg temp/1781169630_3235124_1418276890_dfbe096aa2b1e380607d84dd1db2b2db.jpg temp/1781169630_3235124_1418276831_e6362cab4625da3e91d5ead3568bb889.jpg temp/1781169630_3235124_1418276829_6551d0d76349fe1c7318544cf8bcd1fa.jpg temp/1781169630_3235124_1418276828_037c24a69eea74b8e9b48f7ef487f19e.jpg temp/1781169630_3235124_1418276826_8d22ac361b1ddb824fc1d0b133089882.jpg temp/1781169630_3235124_1418276824_60619d8efb2ad8acc19944fbdd929269.jpg temp/1781169630_3235124_1418276821_0c390311ccd492c80703504662e3818f.jpg temp/1781169630_3235124_1418275683_c6effa8736076befa77ebb7da49c46aa.jpg temp/1781169630_3235124_1418275682_56f09551c09105d74c835663e318d508.jpg temp/1781169630_3235124_1418275680_1fe277860fab441767411e0912fbc7c8.jpg 116it [00:01, 58.12it/s] 123it [00:02, 58.86it/s] 129it [00:02, 57.97it/s] 135it [00:02, 54.52it/s] 142it [00:02, 57.38it/s] 148it [00:02, 53.68it/s] 154it [00:02, 50.74it/s] 160it [00:02, 47.92it/s] 165it [00:02, 46.99it/s] 170it [00:03, 46.52it/s] 176it [00:03, 49.78it/s] 182it [00:03, 47.56it/s] 188it [00:03, 50.29it/s] 194it [00:03, 49.15it/s] 200it [00:03, 49.95it/s] 200it [00:03, 54.98it/s]2026-06-11 11:23:44.811831: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2026-06-11 11:23:44.817860: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2026-06-11 11:23:44.989549: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2026-06-11 11:23:44.998872: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2026-06-11 11:23:45.000634: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1781169630_3235124_1418275678_3ee4bebce2d46ca388de8b5b0298ce56.jpg temp/1781169630_3235124_1418275676_275d44404c11b3e5eb3d6494624ff8f3.jpg temp/1781169630_3235124_1418275672_965ccdffcb787f66180687705f7cb743.jpg temp/1781169630_3235124_1418275604_983ae181e9663c9b9cc7da2918644571.jpg temp/1781169630_3235124_1418275602_94092c2410854bd4fed60172d3c52ea9.jpg temp/1781169630_3235124_1418275599_21eec17f18791c91c2c607be6c8cf283.jpg temp/1781169630_3235124_1418275596_eea574a441036243cc9b56114af242b2.jpg temp/1781169630_3235124_1418275594_32d3ef82c8dd1b92abe241e6cc61a707.jpg temp/1781169630_3235124_1418275593_dae4b768f75c32c44d05e8e835d93b1f.jpg temp/1781169630_3235124_1418275513_2e9a130045f9e3c2071a663a8de8304e.jpg temp/1781169630_3235124_1418275466_13d01fe8f425a019d7399871f18bc72c.jpg temp/1781169630_3235124_1418275225_1780ebf40fea1c5c6bc6effc188d3128.jpg temp/1781169630_3235124_1418275223_c2f1d8e07f7387a2886bdc003ddcbc4a.jpg temp/1781169630_3235124_1418275221_0271fd3fa8474a085edbb78a5da842a3.jpg temp/1781169630_3235124_1418275220_04523d738baabf9141b5e84e400864e2.jpg temp/1781169630_3235124_1418273693_2b642fc9893ff12581ea4d31b527571f.jpg temp/1781169630_3235124_1418273690_705158a255cef3d5ce9a99ec8fa8881d.jpg temp/1781169630_3235124_1418273689_29e8f223324a0af321989512833c83cc.jpg temp/1781169630_3235124_1418273688_039e28f2d3db6368b7bdd5159d8b78b0.jpg temp/1781169630_3235124_1418273687_897da10c58f450c53aad856d18f90a0a.jpg temp/1781169630_3235124_1418273686_c6b850b2f7c7faddaf7cd3ca614711b8.jpg temp/1781169630_3235124_1418273585_9f9df4296cd047530e7e672f74d79b3d.jpg temp/1781169630_3235124_1418273584_9a5d98e184d4a061a8c1370d1745d942.jpg temp/1781169630_3235124_1418273583_6bc2be531b30c8ffebcb5e7348874864.jpg temp/1781169630_3235124_1418273582_90977367c595490ce4d12dd1c6094a96.jpg temp/1781169630_3235124_1418273581_15e0b25834e1bf61aa5ac3c5f84a81fd.jpg temp/1781169630_3235124_1418273580_5d6c2261a29b66f9fc145a307b0325ef.jpg temp/1781169630_3235124_1418273454_60f47732fa2926e72f7a3255e9776f9b.jpg temp/1781169630_3235124_1418273452_23fec957cf2c1e076fc3b98e1e939595.jpg temp/1781169630_3235124_1418273450_5ae83a25c90a6f38fbf215b5fde8e87e.jpg temp/1781169630_3235124_1418273448_190c377dc34e2e728ab11933849f7839.jpg temp/1781169630_3235124_1418273445_ef89807def2412910470f15866f64db5.jpg temp/1781169630_3235124_1418273443_df9fe7b70164fd10bd5bb76e8a1003e0.jpg temp/1781169630_3235124_1418273249_5f80e2673f3fb6f82eabef553f450c68.jpg temp/1781169630_3235124_1418273248_ac44a104f80722f04270d1920489c36e.jpg temp/1781169630_3235124_1418273247_4f0d99055bcd6ab8fb1ac31ddda99840.jpg temp/1781169630_3235124_1418273246_26212c6ee562b0849acf8f1bd42d7740.jpg temp/1781169630_3235124_1418273245_858b3827681e72be2fff5fb9d34fe108.jpg temp/1781169630_3235124_1418273244_1c51d7cab9fe1b565a71e3e1e5e34cfb.jpg temp/1781169630_3235124_1418273164_a3df05f69ffbf7539a9ade442e1cec7c.jpg temp/1781169630_3235124_1418273163_c60c3a0ab5fc1b3cb19e5a936fc3419f.jpg temp/1781169630_3235124_1418273161_8265d18603ed9c8f2ade12981f72d4fe.jpg temp/1781169630_3235124_1418273159_d47369869e563f7d4f7a2caf7bce8f51.jpg temp/1781169630_3235124_1418273157_c170e539500f596e24187a00d19f40ad.jpg temp/1781169630_3235124_1418273154_e3b21a9936ed72ad314929ad1550ad05.jpg temp/1781169630_3235124_1418271864_0fd4a4b07c375a817d981ae230e089b4.jpg temp/1781169630_3235124_1418271863_28a3002e27eacf5a9a6708ba75b597e0.jpg temp/1781169630_3235124_1418271862_fc08277c5cb21782c1e0c0f43ea3bbe4.jpg temp/1781169630_3235124_1418271860_7a36db0466817ad67a38b28cf8a45e1a.jpg temp/1781169630_3235124_1418271858_2360fffdc7c1669826a9fea2697a823f.jpg temp/1781169630_3235124_1418271856_7cd0a0a02f2aa5b4a89a3dd17c062ef8.jpg temp/1781169630_3235124_1418270790_45e9f20d304a030468e3adf28c46c66b.jpg temp/1781169630_3235124_1418270787_546102ec0337df562109fc84c362ff53.jpg temp/1781169630_3235124_1418270785_ecc2fa620f2eca3e5baffa4cfd290bcc.jpg temp/1781169630_3235124_1418270783_22c8177edbe71b331e164f796cec8736.jpg temp/1781169630_3235124_1418270782_71fe23311eb4175eb1b442142859d03c.jpg temp/1781169630_3235124_1418270781_333a1978ae067aec699e08fd4a1969cd.jpg temp/1781169630_3235124_1418270661_11465dd0a8ef8a9d2c833081eae90432.jpg temp/1781169630_3235124_1418270660_e399556d1716ac20bb3fc72f167f8df6.jpg temp/1781169630_3235124_1418270659_b394fe6b9b9ed94489af1d971d356922.jpg temp/1781169630_3235124_1418270658_0a4860be7ec5500379a8027d73cca8a0.jpg temp/1781169630_3235124_1418270657_9532db491f494d4bbdc13f09c05c55d7.jpg temp/1781169630_3235124_1418270655_30f9581b2ecc0b02e95ec68ff174661d.jpg temp/1781169630_3235124_1418270591_6a24a66811b5bbb1eeb26716dc3d88ab.jpg temp/1781169630_3235124_1418270589_9bde4319c950e7e7aeb268a2654e11ae.jpg temp/1781169630_3235124_1418270587_154f4fd8e3021554d0e9be46bda9fa17.jpg temp/1781169630_3235124_1418270585_10447c2f59b42eba9d5496f8b72de8ea.jpg temp/1781169630_3235124_1418270584_68a7ae9f20530e33c35c9cc5d16ef169.jpg temp/1781169630_3235124_1418270582_c6153a45c8a28d8664d273c1996f5fe5.jpg temp/1781169630_3235124_1418270499_e3d5eab92d0303788db2e1628ca10bf7.jpg temp/1781169630_3235124_1418270497_545c166ed4d535e3a30d310f2ab1dbcd.jpg temp/1781169630_3235124_1418270494_da979c618f2c50bc2bcc1f34e66ece31.jpg temp/1781169630_3235124_1418270493_cab6ca73c47171115d6693308972b308.jpg temp/1781169630_3235124_1418270492_df2293e6d7f957a1dbd93e25055b0cbb.jpg temp/1781169630_3235124_1418270491_63275466f8fc0cce8ac1b91c02008feb.jpg temp/1781169630_3235124_1418269636_330e79046648dcf4e7fd8b0f8e4b2de2.jpg temp/1781169630_3235124_1418269632_82f41890c1fe6f5b405133ddc1fd119c.jpg temp/1781169630_3235124_1418269626_395effb2da459de31cecb2c7a2a16bbc.jpg temp/1781169630_3235124_1418269624_d26a5c7f5665e325c36fb01648d24f1a.jpg temp/1781169630_3235124_1418269622_b1278d379f06e446463b2fdfef507df2.jpg temp/1781169630_3235124_1418269619_2c25f451d8af303943b061cb1585e1d2.jpg temp/1781169630_3235124_1418268855_5638e060f2bfdf4a71b5856e7bf92287.jpg temp/1781169630_3235124_1418268853_dae2bbacdfdfb91dfae1a01d365478f1.jpg temp/1781169630_3235124_1418268852_34badb19cb5136f897757381a5e8c09b.jpg temp/1781169630_3235124_1418268091_f12eac06f6f874646c740ea3976e4e0c.jpg temp/1781169630_3235124_1418268081_1774d1a16c919b70cce3b7c7b3a41bd7.jpg temp/1781169630_3235124_1418268080_39229b9fe67165737db43e2078b6d947.jpg Found 200 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2340, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2534, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3171, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1418283852, 1418283851, 1418283792, 1418283791, 1418283790, 1418283785, 1418283784, 1418283782, 1418282386, 1418282385, 1418282384, 1418282382, 1418282379, 1418282378, 1418282333, 1418282332, 1418282331, 1418282330, 1418282329, 1418282328, 1418282250, 1418282248, 1418282246, 1418282244, 1418282241, 1418282239, 1418282161, 1418282160, 1418282159, 1418282158, 1418282157, 1418282156, 1418282111, 1418282110, 1418282108, 1418282106, 1418282105, 1418282103, 1418281767, 1418281766, 1418281765, 1418281763, 1418281761, 1418281759, 1418281676, 1418281674, 1418281672, 1418281670, 1418281668, 1418281667, 1418281623, 1418281622, 1418281621, 1418281620, 1418281619, 1418281617, 1418281575, 1418281573, 1418281570, 1418281568, 1418281566, 1418281565, 1418279482, 1418279480, 1418279470, 1418279468, 1418279466, 1418279465, 1418279426, 1418279424, 1418279422, 1418279421, 1418279419, 1418279416, 1418278622, 1418278621, 1418278620, 1418278619, 1418278617, 1418278615, 1418277058, 1418277056, 1418277055, 1418277054, 1418277053, 1418277052, 1418277013, 1418277012, 1418277011, 1418277010, 1418277009, 1418277008, 1418276963, 1418276961, 1418276959, 1418276957, 1418276955, 1418276952, 1418276902, 1418276900, 1418276896, 1418276894, 1418276892, 1418276890, 1418276831, 1418276829, 1418276828, 1418276826, 1418276824, 1418276821, 1418275683, 1418275682, 1418275680, 1418275678, 1418275676, 1418275672, 1418275604, 1418275602, 1418275599, 1418275596, 1418275594, 1418275593, 1418275513, 1418275466, 1418275225, 1418275223, 1418275221, 1418275220, 1418273693, 1418273690, 1418273689, 1418273688, 1418273687, 1418273686, 1418273585, 1418273584, 1418273583, 1418273582, 1418273581, 1418273580, 1418273454, 1418273452, 1418273450, 1418273448, 1418273445, 1418273443, 1418273249, 1418273248, 1418273247, 1418273246, 1418273245, 1418273244, 1418273164, 1418273163, 1418273161, 1418273159, 1418273157, 1418273154, 1418271864, 1418271863, 1418271862, 1418271860, 1418271858, 1418271856, 1418270790, 1418270787, 1418270785, 1418270783, 1418270782, 1418270781, 1418270661, 1418270660, 1418270659, 1418270658, 1418270657, 1418270655, 1418270591, 1418270589, 1418270587, 1418270585, 1418270584, 1418270582, 1418270499, 1418270497, 1418270494, 1418270493, 1418270492, 1418270491, 1418269636, 1418269632, 1418269626, 1418269624, 1418269622, 1418269619, 1418268855, 1418268853, 1418268852, 1418268091, 1418268081, 1418268080] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 200 time used for this insertion : 0.1856682300567627 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 143.9747977256775 time spend to save output : 0.19474482536315918 total time spend for step 0 : 144.16954255104065 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 14.73user 11.32system 3:20.46elapsed 12%CPU (0avgtext+0avgdata 1744512maxresident)k 4685936inputs+6664120outputs (5297major+648449minor)pagefaults 0swaps