inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 50 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 50 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 50 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 50 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 50 wait 20 seconds 2025-05-08 12:52:35.934603: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-08 12:52:35.944288: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-08 12:52:35.944540: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-08 12:52:35.947574: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-08 12:52:35.949504: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-08 12:52:35.950244: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-08 12:52:35.953418: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-08 12:52:35.954688: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-08 12:52:35.961411: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-08 12:52:35.962461: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-08 12:52:35.962876: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-08 12:52:35.970398: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-08 12:52:35.971738: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f7c68000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-08 12:52:35.971782: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-08 12:52:36.230815: W tensorflow/compiler/xla/service/platform_util.cc:210] unable to create StreamExecutor for CUDA:0: failed initializing StreamExecutor for CUDA device ordinal 0: Internal: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_OUT_OF_MEMORY: out of memory; total memory reported: 11553341440 2025-05-08 12:52:36.231086: I tensorflow/compiler/jit/xla_gpu_device.cc:161] Ignoring visible XLA_GPU_JIT device. Device number is 0, reason: Internal: no supported devices found for platform CUDA 2025-05-08 12:52:36.232119: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-08 12:52:36.232220: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-08 12:52:36.232259: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-08 12:52:36.232289: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-08 12:52:36.232319: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-08 12:52:36.232365: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-08 12:52:36.232394: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-08 12:52:36.232430: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-08 12:52:36.233537: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-08 12:52:36.233603: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 l 3637 free memory gpu now : 50 max_wait_temp : 6 max_wait : 5 ERROR in datou_step_exec, will save and exit ! CUDA runtime implicit initialization on GPU:0 failed. Status: out of memory File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3074, in datou_step_tfhub2 logical_gpus = tf.config.experimental.list_logical_devices('GPU') File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/config.py", line 372, in list_logical_devices return context.context().list_logical_devices(device_type=device_type) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/context.py", line 1251, in list_logical_devices self.ensure_initialized() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/context.py", line 515, in ensure_initialized context_handle = pywrap_tfe.TFE_NewContext(opts) [1356992777, 1356992774, 1356992771, 1356992767, 1356992763, 1356992760] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 6 time used for this insertion : 0.013588666915893555 save_final ERROR in last step tfhub_classification2, CUDA runtime implicit initialization on GPU:0 failed. Status: out of memory time spend for datou_step_exec : 128.8322741985321 time spend to save output : 0.01916337013244629 total time spend for step 0 : 128.85143756866455 need to delete datou_research and reload, so keep current state 1 need to delete datou_research and reload, so keep current state 1 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 3.67user 3.14system 2:12.21elapsed 5%CPU (0avgtext+0avgdata 529588maxresident)k 98864inputs+5560outputs (681major+173231minor)pagefaults 0swaps