2025-05-14 11:30:43.383969: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-14 11:30:44.349337: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2025-05-14 11:30:44.352004: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR temp/1747215027_2456543_1358151374_7568c6ed3e9aef68072b0dad1477820f.jpg temp/1747215027_2456543_1358151306_1bbc84bea88422ae0b82f371f85a6d8a.jpg temp/1747215027_2456543_1358151248_f297fe1f73130940c38002302e43a4f0.jpg temp/1747215027_2456543_1358151176_a779469d581fc8afb50cbcc18b1843a9.jpg temp/1747215027_2456543_1358151087_625217d6a4b130d52cf9fcee79ce6504.jpg temp/1747215027_2456543_1358150986_c5c8e5fed698106e959bae5b585a1607.jpg Found 6 images belonging to 1 classes. begin to do the prediction : ERROR in datou_step_exec, will save and exit ! Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3147, in datou_step_tfhub2 classes, outputs, features = this_model.predict_image_paths(list_paths, keep_aspect_ratio=keep_aspect_ratio, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 288, in predict_image_paths Y_pred, F_pred = self.model.predict(valid_generator, validation_steps) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 88, in _method_wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/keras/engine/training.py", line 1268, in predict tmp_batch_outputs = predict_function(iterator) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __call__ result = self._call(*args, **kwds) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/def_function.py", line 650, in _call return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1661, in _filtered_call return self._call_flat( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 1745, in _call_flat return self._build_call_outputs(self._inference_function.call( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/function.py", line 593, in call outputs = execute.execute( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, [1358151374, 1358151306, 1358151248, 1358151176, 1358151087, 1358150986] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 6 time used for this insertion : 0.01725625991821289 save_final ERROR in last step tfhub_classification2, Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node model/module/StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/conv1/StatefulPartitionedCall/Conv2D}}]] [Op:__inference_predict_function_34490] Function call stack: predict_function time spend for datou_step_exec : 15.160636186599731 time spend to save output : 0.021064281463623047 total time spend for step 0 : 15.181700468063354 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 14.13user 4.53system 0:21.20elapsed 88%CPU (0avgtext+0avgdata 2645852maxresident)k 317552inputs+183976outputs (608major+733637minor)pagefaults 0swaps