python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' --fifo -a 533 -l 100 ' -s datou_current_533 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 4084211 load datou : 533 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 we have a portfolio with more photos than limit : 449>100 please execute split_portfolio.py -i 24288160 -l 100 size over we load limit photo not treated list_input_json: {} Current got : datou_id : 533, datou_cur_ids : ['3193509'] with mtr_portfolio_ids : ['24288160'] and first list_photo_ids : [] new path : /proc/4084211/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : {} origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 100 ; length of list_pids : 100 ; length of list_args : 100 time to download the photos : 23.32054877281189 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 2 step1:tfhub_classification2 Thu Jul 17 10:13:51 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou_step TFHub with tf2 ! nombre de thcls : 3 we are using the classfication for multi_thcl [3513, 3379, 3847] begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 376 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 376 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 376 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 376 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 376 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 376 wait 20 seconds 2025-07-17 10:15:59.527419: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-07-17 10:15:59.559254: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-07-17 10:15:59.559539: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-07-17 10:15:59.561638: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-07-17 10:15:59.563605: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-07-17 10:15:59.563980: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-07-17 10:15:59.566271: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-07-17 10:15:59.567535: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-07-17 10:15:59.571351: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-07-17 10:15:59.572224: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-07-17 10:15:59.572573: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-07-17 10:15:59.603458: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493035000 Hz 2025-07-17 10:15:59.605274: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fb0d4000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-07-17 10:15:59.605332: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-07-17 10:15:59.713047: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x10f2ebb0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-07-17 10:15:59.713080: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-07-17 10:15:59.713905: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-07-17 10:15:59.713980: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-07-17 10:15:59.714009: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-07-17 10:15:59.714035: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-07-17 10:15:59.714061: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-07-17 10:15:59.714086: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-07-17 10:15:59.714111: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-07-17 10:15:59.714137: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-07-17 10:15:59.715166: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-07-17 10:15:59.715228: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-07-17 10:15:59.715893: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-07-17 10:15:59.715918: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-07-17 10:15:59.715932: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-07-17 10:15:59.717008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 376 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3513 To do loadFromThcl(), then load ParamDescType : thcl3513 thcls : [{'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] thcl {'id': 3513, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2_tf', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 4557, 'photo_desc_type': 5767, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'} Update svm_hashtag_type_desc : 5767 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5767, 'Rungis_amount_dechets_fall_2018_v2_tf', 2048, 2048, 'Rungis_amount_dechets_fall_2018_v2_tf', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 3, 16, 15, 52, 10), datetime.datetime(2023, 3, 16, 15, 52, 10)) model_name : Rungis_amount_dechets_fall_2018_v2_tf model_param file didn't exist model_name : Rungis_amount_dechets_fall_2018_v2_tf model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_05102018_Papier_non_papier_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg', 'Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00001', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-07-17 10:16:06.324156: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.324666: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.72G (2921752064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.325141: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2629576704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.325615: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.20G (2366618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.326085: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.98G (2129957120 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.326554: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.79G (1916961536 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.327022: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.61G (1725265408 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.327515: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.45G (1552738816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.327987: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.30G (1397465088 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.328455: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.17G (1257718528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.328922: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.05G (1131946752 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.329390: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 971.56M (1018752256 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.329857: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 874.40M (916877056 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.330326: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 786.96M (825189376 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.330806: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 708.27M (742670592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.331283: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 637.44M (668403712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.331754: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 573.70M (601563392 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.332223: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 516.33M (541407232 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.332693: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 464.69M (487266560 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.333163: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 418.22M (438540032 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.333633: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 376.40M (394686208 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.334103: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 338.76M (355217664 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.334573: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 304.89M (319696128 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.335043: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 274.40M (287726592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.335571: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 246.96M (258953984 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.336052: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 222.26M (233058816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.336519: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 200.04M (209753088 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.336952: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 180.03M (188777984 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.337409: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 162.03M (169900288 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.337879: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 145.83M (152910336 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.338349: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 131.24M (137619456 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.338821: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 118.12M (123857664 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.339322: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 106.31M (111472128 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.339799: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 95.68M (100325120 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.340271: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 86.11M (90292736 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.340744: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 77.50M (81263616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.341217: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 69.75M (73137408 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.341692: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 62.77M (65823744 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.342138: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 56.50M (59241472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.342584: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 50.85M (53317376 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.343033: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 45.76M (47985664 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.343514: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 41.19M (43187200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.343997: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 37.07M (38868480 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.344480: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 33.36M (34981632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.344941: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 30.02M (31483648 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.345398: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 27.02M (28335360 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.345856: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 24.32M (25501952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.346319: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 21.89M (22951936 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.346785: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 19.70M (20656896 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.347265: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 17.73M (18591232 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.347758: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 15.96M (16732160 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.348246: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 14.36M (15058944 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.349074: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 12.92M (13553152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.350006: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 11.63M (12197888 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.350890: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 10.47M (10978304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.351816: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 9.42M (9880576 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.352563: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 8.48M (8892672 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.353096: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 7.63M (8003584 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.353627: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 6.87M (7203328 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.354158: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 6.18M (6483200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.354713: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 5.56M (5835008 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.355252: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 5.01M (5251584 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.355815: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.51M (4726528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.356376: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.06M (4253952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.356962: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.65M (3828736 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.357544: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.29M (3446016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.358126: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.96M (3101440 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.358708: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.66M (2791424 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.359314: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.40M (2512384 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.359942: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.16M (2261248 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.360739: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.94M (2035200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.361488: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.75M (1831680 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.362239: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.57M (1648640 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.362989: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.42M (1483776 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.363760: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.27M (1335552 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.364514: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.15M (1202176 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.365265: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.03M (1082112 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.366015: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 951.2K (974080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.366766: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 856.2K (876800 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.367545: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 770.8K (789248 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.368349: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 693.8K (710400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.369104: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 624.5K (639488 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.369854: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 562.2K (575744 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.370612: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 506.2K (518400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.371380: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 455.8K (466688 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.372183: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 410.2K (420096 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.372951: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 369.2K (378112 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.373706: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 332.5K (340480 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.374505: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 299.2K (306432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.375272: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 269.5K (275968 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.376078: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 242.8K (248576 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.376854: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 218.5K (223744 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.377604: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 196.8K (201472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.378355: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 177.2K (181504 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.379105: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 159.8K (163584 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.379907: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 144.0K (147456 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.380710: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 129.8K (132864 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.381507: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 117.0K (119808 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.382258: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 105.5K (108032 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.383009: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 95.0K (97280 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.383804: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 85.5K (87552 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.384603: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 77.0K (78848 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.385354: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 69.5K (71168 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.386104: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 62.8K (64256 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.386865: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 56.5K (57856 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.387682: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 51.0K (52224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.388493: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 46.0K (47104 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.389244: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 41.5K (42496 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.389995: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 37.5K (38400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.390746: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 33.8K (34560 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.391524: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 30.5K (31232 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.392327: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 27.5K (28160 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.393089: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 24.8K (25344 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.393839: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 22.5K (23040 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.394589: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 20.2K (20736 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.395356: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 18.2K (18688 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.396159: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 16.5K (16896 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.396932: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 15.0K (15360 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.397682: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 13.5K (13824 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.398432: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 12.2K (12544 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.399197: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 11.2K (11520 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.400001: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 10.2K (10496 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.400785: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 9.2K (9472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.401536: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 8.5K (8704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.402286: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 7.8K (7936 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.403036: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 7.0K (7168 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.403833: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 6.5K (6656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.404636: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 6.0K (6144 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.405437: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 5.5K (5632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.406199: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 5.0K (5120 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.406960: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.5K (4608 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.407753: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.2K (4352 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.408556: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.0K (4096 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.409327: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.8K (3840 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.410128: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.5K (3584 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.410930: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.2K (3328 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.411747: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.0K (3072 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.412550: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.8K (2816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.413301: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.5K (2560 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.414057: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.414811: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.415594: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.416397: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.417173: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.417974: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.418761: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.419541: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.420344: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.421110: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.421860: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.422610: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.423388: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.424193: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.424970: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.425733: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.426489: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.427249: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.428052: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.428837: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.429597: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.430398: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.431213: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.432017: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.432805: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.433556: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.434307: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.435058: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.435855: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.436658: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.437436: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.438237: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.439008: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.439809: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.440612: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.441368: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.442168: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.442944: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.443730: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.444541: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.445301: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.446051: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.446802: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.447578: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.448380: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.449149: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.449905: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.450661: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.451442: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.452247: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.453024: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.453775: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.454558: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.455366: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.456170: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.456972: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.457764: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.458514: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.459328: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.460134: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.460935: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.461737: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.462548: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.463358: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.464161: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.464963: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.465764: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.466564: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.467372: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.468176: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.468976: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.469777: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.470577: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.471394: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.472199: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.473178: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.474021: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.474823: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.475639: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.476442: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.477244: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.478044: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.478845: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.479668: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.480472: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.481272: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.482081: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.482883: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.483741: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.484545: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.485346: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.486146: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.486946: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.487757: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.488572: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.489377: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.490177: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.490978: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.491814: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.492623: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.493423: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.494224: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.495026: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.495844: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.496653: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.497441: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.498241: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.499038: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.499861: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.500664: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.501474: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.502275: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.503075: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.503902: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.504706: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.505507: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.506308: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.507086: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.507915: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.508720: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.509521: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.510287: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.511037: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.511850: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.512669: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.513474: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.514255: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.515034: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.515844: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.516650: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.517437: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.518238: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.519018: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.519823: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.520634: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.521407: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.522207: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.523007: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.523828: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.524634: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.525417: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.526218: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.527002: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.527795: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.528598: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.529398: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.530199: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.530985: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.531788: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.532592: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.533382: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.534183: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.534973: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.535765: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.536569: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.537369: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.538143: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.538893: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.539771: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.540587: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.541366: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.542403: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.543491: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.544613: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.545687: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.546823: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.547962: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.549288: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.550634: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.551984: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.553318: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.554470: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.555572: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.556670: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.557768: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.558679: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.559594: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.560506: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.561441: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.562366: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.563274: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.564198: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.565118: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.565964: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.566848: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.567738: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.568622: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.569460: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.570289: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.571141: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.572040: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.572912: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.573735: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.574558: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.575449: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.576328: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.577208: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.578075: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.578903: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.579767: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.580645: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.581480: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.582305: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.583127: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.584013: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.584886: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.585709: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.586545: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.587389: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.588268: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.589163: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.590780: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.591913: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.592740: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.593532: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.594304: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.595071: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.595881: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.596662: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.597426: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.598186: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.598959: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.599756: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.600575: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.601368: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.2K (2304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.624753: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246388992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:06.625355: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246388992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:16.626368: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246388992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:16.627229: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246388992 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-07-17 10:16:16.627283: W tensorflow/core/common_runtime/bfc_allocator.cc:434] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1B (rounded to 256) Current allocation summary follows. 2025-07-17 10:16:16.627298: I tensorflow/core/common_runtime/bfc_allocator.cc:934] BFCAllocator dump for GPU_0_bfc 2025-07-17 10:16:16.627313: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (256): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-07-17 10:16:16.627326: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (512): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-07-17 10:16:16.627339: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1024): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-07-17 10:16:16.627363: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2048): Total Chunks: 1, Chunks in use: 1. 2.2KiB allocated for chunks. 2.2KiB in use in bin. 1.0KiB client-requested in use in bin. 2025-07-17 10:16:16.627377: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-07-17 10:16:16.627390: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-07-17 10:16:16.627403: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-07-17 10:16:16.627416: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-07-17 10:16:16.627428: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-07-17 10:16:16.627441: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (131072): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-07-17 10:16:16.627454: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-07-17 10:16:16.627467: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (524288): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-07-17 10:16:16.627479: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-07-17 10:16:16.627492: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-07-17 10:16:16.627505: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-07-17 10:16:16.627518: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-07-17 10:16:16.627531: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-07-17 10:16:16.627543: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-07-17 10:16:16.627565: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-07-17 10:16:16.627578: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-07-17 10:16:16.627591: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-07-17 10:16:16.627604: I tensorflow/core/common_runtime/bfc_allocator.cc:957] Bin for 256B was 256B, Chunk State: 2025-07-17 10:16:16.627615: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 2304 2025-07-17 10:16:16.627631: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fb052e00000 of size 2304 next 18446744073709551615 2025-07-17 10:16:16.627642: I tensorflow/core/common_runtime/bfc_allocator.cc:995] Summary of in-use Chunks by size: 2025-07-17 10:16:16.627656: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 2304 totalling 2.2KiB 2025-07-17 10:16:16.627669: I tensorflow/core/common_runtime/bfc_allocator.cc:1002] Sum Total of in-use chunks: 2.2KiB 2025-07-17 10:16:16.627680: I tensorflow/core/common_runtime/bfc_allocator.cc:1004] total_region_allocated_bytes_: 2304 memory_limit_: 3246391296 available bytes: 3246388992 curr_region_allocation_bytes_: 6492782592 2025-07-17 10:16:16.627695: I tensorflow/core/common_runtime/bfc_allocator.cc:1010] Stats: Limit: 3246391296 InUse: 2304 MaxInUse: 2304 NumAllocs: 1 MaxAllocSize: 2304 2025-07-17 10:16:16.627709: W tensorflow/core/common_runtime/bfc_allocator.cc:439] *********************************************xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx local folder : /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Confusion_Matrix.png size_local : 67810 size in s3 : 67810 create time local : 2023-10-30 16:21:37 create time in s3 : 2023-10-30 14:09:29 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_dense.jpg size_local : 73949 size in s3 : 73949 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Precision_Recall_05102018_Papier_non_papier_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg size_local : 85572 size in s3 : 85572 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:39 Precision_Recall_05102018_Papier_non_papier_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg size_local : 72361 size in s3 : 72361 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:37 Precision_Recall_05102018_Papier_non_papier_presque_vide.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg size_local : 83567 size in s3 : 83567 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:48 Precision_Recall_05102018_Papier_non_papier_tres_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg size_local : 71611 size in s3 : 71611 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:29 Precision_Recall_05102018_Papier_non_papier_tres_peu_dense.jpg already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/Result_Summary.txt size_local : 1058 size in s3 : 1058 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:30 Result_Summary.txt already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-10-30 16:21:38 create time in s3 : 2023-10-30 14:09:36 checkpoint already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00001 size_local : 188538519 size in s3 : 188538519 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-10-30 14:09:40 model_checkpoint.ckpt.data-00000-of-00001 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216572 size in s3 : 216572 create time local : 2023-10-30 16:21:41 create time in s3 : 2023-03-16 14:52:09 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-03-16 14:52:07 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_checkpoint.ckpt.index size_local : 28001 size in s3 : 28001 create time local : 2023-10-30 16:21:42 create time in s3 : 2023-10-30 14:09:30 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/Rungis_amount_dechets_fall_2018_v2_tf/model_weights.h5 size_local : 94501976 size in s3 : 94501976 create time local : 2023-10-30 16:21:44 create time in s3 : 2023-10-30 14:09:37 model_weights.h5 already exist and didn't need to update ERROR in datou_step_exec, will save and exit ! Failed copying input tensor from /job:localhost/replica:0/task:0/device:CPU:0 to /job:localhost/replica:0/task:0/device:GPU:0 in order to run LogicalNot: Dst tensor is not initialized. [Op:LogicalNot] File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3139, in datou_step_tfhub2 this_model = model_evaluator(model_name, model_type=model_type, fc_size=fc_size,use_multi_inputs=use_multi_inputs) File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 156, in __init__ self.model, _, _ = create_tfhub_model(module_handle=self.tfhub_module, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 77, in create_tfhub_model hub.KerasLayer(module_handle, trainable=do_fine_tuning, name="module"), File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 152, in __init__ self._func = load_module(handle, tags, self._load_options) File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 421, in load_module return module_v2.load(handle, tags=tags, options=set_load_options) File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/module_v2.py", line 106, in load obj = tf.compat.v1.saved_model.load_v2(module_path, tags=tags) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 578, in load return load_internal(export_dir, tags) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 602, in load_internal loader = loader_cls(object_graph_proto, File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 123, in __init__ self._load_all() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 134, in _load_all self._load_nodes() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 264, in _load_nodes node, setter = self._recreate(proto, node_id) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 370, in _recreate return factory[kind]() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 363, in "variable": lambda: self._recreate_variable(proto.variable), File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 426, in _recreate_variable return variables.Variable( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 261, in __call__ return cls._variable_v2_call(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 243, in _variable_v2_call return previous_getter( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 66, in getter return captured_getter(captured_previous, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 418, in uninitialized_variable_creator return resource_variable_ops.UninitializedVariable(**kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 263, in __call__ return super(VariableMetaclass, cls).__call__(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 1795, in __init__ handle = _variable_handle_from_shape_and_dtype( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 175, in _variable_handle_from_shape_and_dtype math_ops.logical_not(exists), [exists], name="EagerVariableNameReuse") File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gen_math_ops.py", line 5427, in logical_not _ops.raise_from_not_ok_status(e, name) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status six.raise_from(core._status_to_exception(e.code, message), None) File "", line 3, in raise_from [1372382650, 1372382641, 1372382633, 1372382628, 1372382614, 1372382610, 1372381082, 1372380980, 1372380900, 1372380826, 1372380758, 1372380677, 1372379679, 1372379677, 1372379675, 1372379666, 1372379660, 1372379654, 1372379047, 1372379042, 1372379031, 1372379026, 1372379021, 1372379018, 1372378180, 1372378175, 1372378171, 1372378163, 1372378159, 1372378152, 1372377493, 1372377486, 1372377482, 1372377478, 1372377472, 1372377467, 1372371972, 1372371861, 1372371769, 1372371645, 1372371554, 1372371441, 1372365743, 1372365698, 1372365650, 1372365586, 1372365513, 1372365460, 1372361106, 1372361058, 1372361016, 1372360997, 1372360949, 1372360864, 1372356788, 1372355916, 1372355880, 1372355877, 1372355872, 1372355867, 1372355863, 1372355367, 1372355345, 1372355281, 1372355230, 1372355162, 1372355099, 1372354226, 1372354220, 1372354212, 1372354187, 1372354155, 1372354093, 1372353299, 1372353287, 1372353254, 1372353238, 1372353233, 1372353226, 1372352747, 1372352742, 1372352737, 1372352733, 1372352729, 1372352705, 1372352238, 1372352233, 1372352226, 1372352221, 1372352215, 1372352210, 1372351987, 1372351983, 1372351978, 1372351972, 1372351964, 1372351957, 1372351729, 1372351725, 1372351720] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 100 time used for this insertion : 0.061704397201538086 save_final ERROR in last step tfhub_classification2, Failed copying input tensor from /job:localhost/replica:0/task:0/device:CPU:0 to /job:localhost/replica:0/task:0/device:GPU:0 in order to run LogicalNot: Dst tensor is not initialized. [Op:LogicalNot] time spend for datou_step_exec : 144.84425735473633 time spend to save output : 0.07366681098937988 total time spend for step 0 : 144.9179241657257 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! 9.95user 4.90system 2:52.69elapsed 8%CPU (0avgtext+0avgdata 937896maxresident)k 808inputs+1371352outputs (0major+566043minor)pagefaults 0swaps