2025-05-22 13:10:37.631926: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-22 13:10:37.667266: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-22 13:10:37.669938: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f36b0000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-22 13:10:37.669984: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-22 13:10:37.675743: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-22 13:10:37.979197: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x34522a20 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-22 13:10:37.979235: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-22 13:10:37.980185: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-22 13:10:37.982769: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-22 13:10:38.020887: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-22 13:10:38.044614: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-22 13:10:38.048974: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-22 13:10:38.087841: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-22 13:10:38.093970: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-22 13:10:38.159893: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-22 13:10:38.161316: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-22 13:10:38.161854: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-22 13:10:38.163323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-22 13:10:38.163344: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-22 13:10:38.163355: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-22 13:10:38.165064: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3377 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-05-22 13:10:38.613927: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-22 13:10:38.614027: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-22 13:10:38.614050: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-22 13:10:38.614072: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-22 13:10:38.614094: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-22 13:10:38.614114: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-22 13:10:38.614133: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-22 13:10:38.614153: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-22 13:10:38.615326: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-22 13:10:38.616422: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-22 13:10:38.616467: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-22 13:10:38.616489: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-22 13:10:38.616509: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-22 13:10:38.616528: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-22 13:10:38.616547: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-22 13:10:38.616566: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-22 13:10:38.616585: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-22 13:10:38.617691: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-22 13:10:38.617726: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-22 13:10:38.617736: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-22 13:10:38.617745: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-22 13:10:38.618941: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3377 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admi2025-05-22 13:10:56.177824: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-22 13:10:56.368908: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-22 13:10:58.110988: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.111640: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.52G (2705522688 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.112241: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.27G (2434970368 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.112282: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-22 13:10:58.112889: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.112913: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-22 13:10:58.119713: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.119762: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-22 13:10:58.120483: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.120510: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-22 13:10:58.127179: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.127238: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 466.56MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-22 13:10:58.127969: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.127995: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 466.56MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-22 13:10:58.157602: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.157666: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-22 13:10:58.158250: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.158271: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-22 13:10:58.163955: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.164004: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 243.25MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-22 13:10:58.164577: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.164598: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 243.25MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-22 13:10:58.194683: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.195316: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.197026: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.197605: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.235502: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.236121: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.238105: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.238798: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.264047: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.264633: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.266143: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.266704: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.272230: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.272818: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.274491: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.275077: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.280825: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.281442: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.283006: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.283572: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.311334: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.311937: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.312523: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.313095: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.317353: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.317958: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.334484: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.335124: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.335693: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.336253: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.349171: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.349794: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.350356: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.350952: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.355622: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.356254: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.400275: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.400846: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.420011: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.420623: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.424969: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.425686: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.426398: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.427117: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.428076: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.428799: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.442871: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.443459: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.444044: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.444615: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.445186: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.445761: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.446334: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.446915: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.80G (3006136320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-22 13:10:58.482910: E tensorflow/stream_executor/cuda/cuda_driver.cc:910] failed to synchronize the stop event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-05-22 13:10:58.482985: E tensorflow/stream_executor/gpu/gpu_timer.cc:55] Internal: Error destroying CUDA event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-05-22 13:10:58.482996: E tensorflow/stream_executor/gpu/gpu_timer.cc:60] Internal: Error destroying CUDA event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-05-22 13:10:58.483057: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 8B (8 bytes) from device: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-05-22 13:10:58.483070: E tensorflow/stream_executor/stream.cc:5485] Internal: Failed to enqueue async memset operation: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-05-22 13:10:58.483089: W tensorflow/core/kernels/gpu_utils.cc:69] Failed to check cudnn convolutions for out-of-bounds reads and writes with an error message: 'Failed to load in-memory CUBIN: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered'; skipping this check. This only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-05-22 13:10:58.483099: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 8B (8 bytes) from device: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-05-22 13:10:58.483111: I tensorflow/stream_executor/stream.cc:4963] [stream=0x359d4020,impl=0x359d3010] did not memzero GPU location; source: 0x7f35127fa020 2025-05-22 13:10:58.483404: F ./tensorflow/core/kernels/conv_2d_gpu.h:453] Non-OK-status: GpuLaunchKernel(ShuffleInTensor3Simple, config.block_count, config.thread_per_block, 0, d.stream(), config.virtual_thread_count, in.data(), combined_dims, out.data()) status: Internal: an illegal memory access was encountered max_time_sub_proc : 3600 Catched exception ! Connect or reconnect ! in case -12 caffe_path_current : About to save ! 2 After save, about to update current ! 13.53user 13.65system 1:00:10elapsed 0%CPU (0avgtext+0avgdata 3442456maxresident)k 1935336inputs+56168outputs (16570major+1067571minor)pagefaults 0swaps