python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 20 -a ' -a 3318 ' -s datou_3318 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 1427043 load datou : 3318 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5275, 'learn_RUBBIA_REFUS_AMIENS_23', 16384, 25088, 'learn_RUBBIA_REFUS_AMIENS_23', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 23, 14, 19, 39), datetime.datetime(2021, 4, 23, 14, 19, 39)) load thcls load THCL from format json or kwargs add thcl : 2847 in CacheModelConfig load pdts add pdt : 5275 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 list_input_json: [] Date 2026-02-27 17:20:29.454046 Current got : datou_id : 3318, datou_cur_ids : ['4353714'] with mtr_portfolio_ids : ['30541517'] and first list_photo_ids : [] new path : /proc/1427043/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, rle_unique_nms_with_priority, ventilate_hashtags_in_portfolio, final, blur_detection, brightness, velours_tree, send_mail_cod, split_time_score over limit max, limiting to limit_max 40 list_input_json : [] origin We have 1 , BFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 9 ; length of list_pids : 9 ; length of list_args : 9 time to download the photos : 1.6795415878295898 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 10 step1:mask_detect Fri Feb 27 17:20:31 2026 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3000 l 3632 free memory gpu now : 776 wait 20 seconds l 3637 free memory gpu now : 776 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2026-02-27 17:20:53.749165: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2026-02-27 17:20:53.774438: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493010000 Hz 2026-02-27 17:20:53.776444: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f48d8000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2026-02-27 17:20:53.776490: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2026-02-27 17:20:53.779978: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2026-02-27 17:20:53.919446: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3db9cfe0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2026-02-27 17:20:53.919504: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2026-02-27 17:20:53.920414: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2026-02-27 17:20:53.920807: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2026-02-27 17:20:53.923664: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2026-02-27 17:20:53.926237: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2026-02-27 17:20:53.926754: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2026-02-27 17:20:53.929834: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2026-02-27 17:20:53.931564: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2026-02-27 17:20:53.935725: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2026-02-27 17:20:53.936593: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2026-02-27 17:20:53.936660: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2026-02-27 17:20:53.937116: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2026-02-27 17:20:53.937132: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2026-02-27 17:20:53.937150: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2026-02-27 17:20:53.937913: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 186 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2026-02-27 17:20:54.197588: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2026-02-27 17:20:54.197683: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2026-02-27 17:20:54.197711: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2026-02-27 17:20:54.197736: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2026-02-27 17:20:54.197761: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2026-02-27 17:20:54.197786: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2026-02-27 17:20:54.197810: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2026-02-27 17:20:54.197834: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2026-02-27 17:20:54.198989: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2026-02-27 17:20:54.200304: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2026-02-27 17:20:54.200363: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2026-02-27 17:20:54.200390: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2026-02-27 17:20:54.200414: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2026-02-27 17:20:54.200438: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2026-02-27 17:20:54.200462: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2026-02-27 17:20:54.200485: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2026-02-27 17:20:54.200509: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2026-02-27 17:20:54.201555: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2026-02-27 17:20:54.201595: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2026-02-27 17:20:54.201610: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2026-02-27 17:20:54.201624: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2026-02-27 17:20:54.202765: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 186 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) 2026-02-27 17:20:55.392704: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 64.00M (67108864 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.393221: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 57.60M (60398080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.393700: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 51.84M (54358272 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.394146: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 46.66M (48922624 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.394631: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 41.99M (44030464 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.395113: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 37.79M (39627520 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.395593: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 34.01M (35664896 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.396051: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 30.61M (32098560 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.396503: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 27.55M (28888832 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.396956: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 24.79M (26000128 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.397412: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 22.32M (23400192 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.397870: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 20.08M (21060352 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.398339: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 18.08M (18954496 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.398834: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 16.27M (17059072 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.399323: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 14.64M (15353344 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.399814: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 13.18M (13818112 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.400314: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 11.86M (12436480 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.400807: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 10.67M (11192832 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.401302: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 9.61M (10073600 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.401813: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 8.65M (9066240 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.402348: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 7.78M (8159744 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.402879: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 7.00M (7343872 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.403408: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 6.30M (6609664 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.403961: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 5.67M (5948928 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.404479: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 5.11M (5354240 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.404996: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.60M (4818944 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.405514: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.14M (4337152 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.437850: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 119.96M (125792256 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:20:55.438346: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 119.96M (125792256 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:21:05.439353: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 119.96M (125792256 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:21:05.440179: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 119.96M (125792256 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2026-02-27 17:21:05.440213: W tensorflow/core/common_runtime/bfc_allocator.cc:434] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.25MiB (rounded to 2359296) Current allocation summary follows. 2026-02-27 17:21:05.440235: I tensorflow/core/common_runtime/bfc_allocator.cc:934] BFCAllocator dump for GPU_0_bfc 2026-02-27 17:21:05.440253: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (256): Total Chunks: 60, Chunks in use: 60. 15.0KiB allocated for chunks. 15.0KiB in use in bin. 8.8KiB client-requested in use in bin. 2026-02-27 17:21:05.440269: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (512): Total Chunks: 40, Chunks in use: 40. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin. 2026-02-27 17:21:05.440285: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1024): Total Chunks: 136, Chunks in use: 136. 136.2KiB allocated for chunks. 136.2KiB in use in bin. 136.0KiB client-requested in use in bin. 2026-02-27 17:21:05.440305: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2048): Total Chunks: 25, Chunks in use: 25. 50.0KiB allocated for chunks. 50.0KiB in use in bin. 50.0KiB client-requested in use in bin. 2026-02-27 17:21:05.440321: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4096): Total Chunks: 60, Chunks in use: 60. 245.5KiB allocated for chunks. 245.5KiB in use in bin. 240.0KiB client-requested in use in bin. 2026-02-27 17:21:05.440335: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2026-02-27 17:21:05.440350: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16384): Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin. 2026-02-27 17:21:05.440365: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (32768): Total Chunks: 1, Chunks in use: 1. 36.8KiB allocated for chunks. 36.8KiB in use in bin. 36.8KiB client-requested in use in bin. 2026-02-27 17:21:05.440380: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (65536): Total Chunks: 7, Chunks in use: 6. 472.8KiB allocated for chunks. 384.0KiB in use in bin. 384.0KiB client-requested in use in bin. 2026-02-27 17:21:05.440395: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (131072): Total Chunks: 4, Chunks in use: 4. 560.0KiB allocated for chunks. 560.0KiB in use in bin. 560.0KiB client-requested in use in bin. 2026-02-27 17:21:05.440410: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (262144): Total Chunks: 8, Chunks in use: 7. 2.16MiB allocated for chunks. 1.91MiB in use in bin. 1.75MiB client-requested in use in bin. 2026-02-27 17:21:05.440438: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (524288): Total Chunks: 8, Chunks in use: 6. 4.95MiB allocated for chunks. 3.44MiB in use in bin. 3.25MiB client-requested in use in bin. 2026-02-27 17:21:05.440454: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1048576): Total Chunks: 24, Chunks in use: 22. 27.12MiB allocated for chunks. 24.88MiB in use in bin. 22.00MiB client-requested in use in bin. 2026-02-27 17:21:05.440469: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2097152): Total Chunks: 13, Chunks in use: 13. 30.97MiB allocated for chunks. 30.97MiB in use in bin. 29.00MiB client-requested in use in bin. 2026-02-27 17:21:05.440483: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2026-02-27 17:21:05.440496: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2026-02-27 17:21:05.440509: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2026-02-27 17:21:05.440522: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2026-02-27 17:21:05.440535: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2026-02-27 17:21:05.440548: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2026-02-27 17:21:05.440561: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2026-02-27 17:21:05.440574: I tensorflow/core/common_runtime/bfc_allocator.cc:957] Bin for 2.25MiB was 2.00MiB, Chunk State: 2026-02-27 17:21:05.440586: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 33554432 2026-02-27 17:21:05.440603: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f0000000 of size 2359296 next 245 2026-02-27 17:21:05.440615: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f0240000 of size 1048576 next 269 2026-02-27 17:21:05.440627: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f46f0340000 of size 1310720 next 263 2026-02-27 17:21:05.440638: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f0480000 of size 2359296 next 262 2026-02-27 17:21:05.440650: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f06c0000 of size 1048576 next 287 2026-02-27 17:21:05.440662: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f07c0000 of size 1310720 next 281 2026-02-27 17:21:05.440674: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f0900000 of size 2359296 next 280 2026-02-27 17:21:05.440686: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f0b40000 of size 1048576 next 304 2026-02-27 17:21:05.440697: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f0c40000 of size 1310720 next 298 2026-02-27 17:21:05.440709: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f0d80000 of size 2359296 next 297 2026-02-27 17:21:05.440721: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f0fc0000 of size 1048576 next 322 2026-02-27 17:21:05.440732: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f10c0000 of size 1310720 next 316 2026-02-27 17:21:05.440752: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f1200000 of size 2359296 next 315 2026-02-27 17:21:05.440764: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f1440000 of size 1048576 next 340 2026-02-27 17:21:05.440776: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f1540000 of size 1310720 next 334 2026-02-27 17:21:05.440788: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f1680000 of size 2359296 next 333 2026-02-27 17:21:05.440799: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f18c0000 of size 1048576 next 358 2026-02-27 17:21:05.440811: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f19c0000 of size 1310720 next 352 2026-02-27 17:21:05.440822: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f1b00000 of size 2359296 next 351 2026-02-27 17:21:05.440834: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f1d40000 of size 1048576 next 375 2026-02-27 17:21:05.440846: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f1e40000 of size 1835008 next 18446744073709551615 2026-02-27 17:21:05.440857: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 16777216 2026-02-27 17:21:05.440869: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f2000000 of size 2359296 next 181 2026-02-27 17:21:05.440881: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f2240000 of size 2097152 next 196 2026-02-27 17:21:05.440893: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f2440000 of size 1048576 next 216 2026-02-27 17:21:05.440904: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f2540000 of size 1310720 next 209 2026-02-27 17:21:05.440916: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f2680000 of size 2359296 next 208 2026-02-27 17:21:05.440928: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f28c0000 of size 1048576 next 234 2026-02-27 17:21:05.440939: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f29c0000 of size 1310720 next 228 2026-02-27 17:21:05.440951: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f2b00000 of size 2359296 next 227 2026-02-27 17:21:05.440963: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f2d40000 of size 2883584 next 18446744073709551615 2026-02-27 17:21:05.440974: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 3903488 2026-02-27 17:21:05.440986: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f3000000 of size 3903488 next 18446744073709551615 2026-02-27 17:21:05.440997: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 2097152 2026-02-27 17:21:05.441010: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4600000 of size 147456 next 55 2026-02-27 17:21:05.441022: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4624000 of size 65536 next 78 2026-02-27 17:21:05.441034: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4634000 of size 4096 next 191 2026-02-27 17:21:05.441046: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4635000 of size 4096 next 192 2026-02-27 17:21:05.441057: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4636000 of size 4096 next 193 2026-02-27 17:21:05.441069: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4637000 of size 256 next 194 2026-02-27 17:21:05.441081: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4637100 of size 256 next 195 2026-02-27 17:21:05.441092: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4637200 of size 4096 next 197 2026-02-27 17:21:05.441104: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4638200 of size 4096 next 198 2026-02-27 17:21:05.441115: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4639200 of size 4096 next 199 2026-02-27 17:21:05.441135: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f463a200 of size 4096 next 200 2026-02-27 17:21:05.441147: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f463b200 of size 4096 next 201 2026-02-27 17:21:05.441159: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f463c200 of size 1024 next 202 2026-02-27 17:21:05.441171: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f463c600 of size 1024 next 204 2026-02-27 17:21:05.441182: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f463ca00 of size 1024 next 205 2026-02-27 17:21:05.441194: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f463ce00 of size 1024 next 206 2026-02-27 17:21:05.441205: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f463d200 of size 1024 next 207 2026-02-27 17:21:05.441217: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f463d600 of size 1024 next 210 2026-02-27 17:21:05.441228: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f463da00 of size 1024 next 211 2026-02-27 17:21:05.441240: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f463de00 of size 1024 next 212 2026-02-27 17:21:05.441251: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f463e200 of size 1024 next 213 2026-02-27 17:21:05.441263: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f463e600 of size 1024 next 214 2026-02-27 17:21:05.441275: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f463ea00 of size 4096 next 215 2026-02-27 17:21:05.441286: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f463fa00 of size 4096 next 217 2026-02-27 17:21:05.441298: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4640a00 of size 4096 next 218 2026-02-27 17:21:05.441309: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4641a00 of size 4096 next 219 2026-02-27 17:21:05.441321: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4642a00 of size 4096 next 220 2026-02-27 17:21:05.441332: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4643a00 of size 1024 next 221 2026-02-27 17:21:05.441344: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4643e00 of size 1024 next 223 2026-02-27 17:21:05.441356: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4644200 of size 1024 next 224 2026-02-27 17:21:05.441367: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4644600 of size 1024 next 225 2026-02-27 17:21:05.441379: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4644a00 of size 1024 next 226 2026-02-27 17:21:05.441390: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4644e00 of size 1024 next 229 2026-02-27 17:21:05.441402: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4645200 of size 1024 next 230 2026-02-27 17:21:05.441413: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4645600 of size 1024 next 231 2026-02-27 17:21:05.441425: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4645a00 of size 1024 next 232 2026-02-27 17:21:05.441436: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4645e00 of size 1024 next 233 2026-02-27 17:21:05.441448: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4646200 of size 7680 next 72 2026-02-27 17:21:05.441460: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4648000 of size 147456 next 71 2026-02-27 17:21:05.441472: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f466c000 of size 131072 next 87 2026-02-27 17:21:05.441484: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f468c000 of size 4096 next 289 2026-02-27 17:21:05.441495: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f468d000 of size 4096 next 290 2026-02-27 17:21:05.441514: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f468e000 of size 4096 next 291 2026-02-27 17:21:05.441526: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f468f000 of size 1024 next 292 2026-02-27 17:21:05.441538: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f468f400 of size 1024 next 293 2026-02-27 17:21:05.441550: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f468f800 of size 1024 next 294 2026-02-27 17:21:05.441561: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f468fc00 of size 1024 next 295 2026-02-27 17:21:05.441573: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4690000 of size 1024 next 296 2026-02-27 17:21:05.441584: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4690400 of size 1024 next 299 2026-02-27 17:21:05.441596: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4690800 of size 1024 next 300 2026-02-27 17:21:05.441607: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4690c00 of size 1024 next 301 2026-02-27 17:21:05.441619: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4691000 of size 1024 next 302 2026-02-27 17:21:05.441631: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4691400 of size 1024 next 303 2026-02-27 17:21:05.441642: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4691800 of size 4096 next 305 2026-02-27 17:21:05.441654: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4692800 of size 4096 next 306 2026-02-27 17:21:05.441665: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4693800 of size 4096 next 307 2026-02-27 17:21:05.441677: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4694800 of size 4096 next 308 2026-02-27 17:21:05.441688: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4695800 of size 4096 next 309 2026-02-27 17:21:05.441700: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4696800 of size 1024 next 310 2026-02-27 17:21:05.441711: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4696c00 of size 1024 next 311 2026-02-27 17:21:05.441723: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4697000 of size 1024 next 312 2026-02-27 17:21:05.441735: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4697400 of size 1024 next 313 2026-02-27 17:21:05.441746: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4697800 of size 1024 next 314 2026-02-27 17:21:05.441758: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4697c00 of size 1024 next 317 2026-02-27 17:21:05.441769: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4698000 of size 1024 next 318 2026-02-27 17:21:05.441781: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4698400 of size 1024 next 319 2026-02-27 17:21:05.441792: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4698800 of size 1024 next 320 2026-02-27 17:21:05.441804: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4698c00 of size 1024 next 321 2026-02-27 17:21:05.441816: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4699000 of size 4096 next 323 2026-02-27 17:21:05.441827: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f469a000 of size 4096 next 324 2026-02-27 17:21:05.441839: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f469b000 of size 4096 next 325 2026-02-27 17:21:05.441850: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f469c000 of size 4096 next 326 2026-02-27 17:21:05.441862: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f469d000 of size 4096 next 327 2026-02-27 17:21:05.441873: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f469e000 of size 1024 next 328 2026-02-27 17:21:05.441885: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f469e400 of size 1024 next 329 2026-02-27 17:21:05.441904: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f469e800 of size 1024 next 330 2026-02-27 17:21:05.441916: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f469ec00 of size 1024 next 331 2026-02-27 17:21:05.441928: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f469f000 of size 1024 next 332 2026-02-27 17:21:05.441939: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f469f400 of size 1024 next 335 2026-02-27 17:21:05.441951: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f469f800 of size 1024 next 336 2026-02-27 17:21:05.441962: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f469fc00 of size 1024 next 337 2026-02-27 17:21:05.441974: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46a0000 of size 1024 next 338 2026-02-27 17:21:05.441985: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46a0400 of size 1024 next 339 2026-02-27 17:21:05.441997: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46a0800 of size 4096 next 341 2026-02-27 17:21:05.442008: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46a1800 of size 4096 next 342 2026-02-27 17:21:05.442020: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46a2800 of size 4096 next 343 2026-02-27 17:21:05.442031: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46a3800 of size 4096 next 344 2026-02-27 17:21:05.442043: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46a4800 of size 4096 next 345 2026-02-27 17:21:05.442054: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46a5800 of size 1024 next 346 2026-02-27 17:21:05.442066: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46a5c00 of size 1024 next 347 2026-02-27 17:21:05.442078: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46a6000 of size 1024 next 348 2026-02-27 17:21:05.442089: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46a6400 of size 1024 next 349 2026-02-27 17:21:05.442101: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46a6800 of size 1024 next 350 2026-02-27 17:21:05.442112: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46a6c00 of size 1024 next 353 2026-02-27 17:21:05.442124: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46a7000 of size 1024 next 354 2026-02-27 17:21:05.442135: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46a7400 of size 1024 next 355 2026-02-27 17:21:05.442147: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46a7800 of size 1024 next 356 2026-02-27 17:21:05.442158: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46a7c00 of size 1024 next 357 2026-02-27 17:21:05.442170: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46a8000 of size 4096 next 359 2026-02-27 17:21:05.442181: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46a9000 of size 4096 next 360 2026-02-27 17:21:05.442193: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46aa000 of size 4096 next 361 2026-02-27 17:21:05.442204: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46ab000 of size 4096 next 362 2026-02-27 17:21:05.442216: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46ac000 of size 4096 next 363 2026-02-27 17:21:05.442227: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46ad000 of size 1024 next 364 2026-02-27 17:21:05.442239: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46ad400 of size 1024 next 365 2026-02-27 17:21:05.442250: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46ad800 of size 1024 next 366 2026-02-27 17:21:05.442262: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46adc00 of size 1024 next 367 2026-02-27 17:21:05.442300: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46ae000 of size 1024 next 368 2026-02-27 17:21:05.442316: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46ae400 of size 1024 next 370 2026-02-27 17:21:05.442327: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46ae800 of size 1024 next 371 2026-02-27 17:21:05.442339: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46aec00 of size 1024 next 372 2026-02-27 17:21:05.442350: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46af000 of size 1024 next 373 2026-02-27 17:21:05.442362: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46af400 of size 1024 next 374 2026-02-27 17:21:05.442373: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46af800 of size 4096 next 376 2026-02-27 17:21:05.442385: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46b0800 of size 4096 next 377 2026-02-27 17:21:05.442396: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46b1800 of size 4096 next 378 2026-02-27 17:21:05.442408: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46b2800 of size 4096 next 379 2026-02-27 17:21:05.442419: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46b3800 of size 4096 next 380 2026-02-27 17:21:05.442431: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46b4800 of size 1024 next 381 2026-02-27 17:21:05.442442: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46b4c00 of size 1024 next 382 2026-02-27 17:21:05.442454: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46b5000 of size 1024 next 383 2026-02-27 17:21:05.442465: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46b5400 of size 1024 next 384 2026-02-27 17:21:05.442477: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46b5800 of size 1024 next 385 2026-02-27 17:21:05.442488: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46b5c00 of size 256 next 386 2026-02-27 17:21:05.442500: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f46f46b5d00 of size 90880 next 103 2026-02-27 17:21:05.442512: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f46cc000 of size 262144 next 102 2026-02-27 17:21:05.442523: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f46f470c000 of size 999424 next 18446744073709551615 2026-02-27 17:21:05.442535: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 4194304 2026-02-27 17:21:05.442547: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4800000 of size 589824 next 96 2026-02-27 17:21:05.442558: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4890000 of size 262144 next 117 2026-02-27 17:21:05.442570: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f46f48d0000 of size 262144 next 112 2026-02-27 17:21:05.442582: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4910000 of size 524288 next 111 2026-02-27 17:21:05.442593: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4990000 of size 262144 next 135 2026-02-27 17:21:05.442605: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f49d0000 of size 327680 next 124 2026-02-27 17:21:05.442617: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4a20000 of size 589824 next 123 2026-02-27 17:21:05.442628: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4ab0000 of size 262144 next 152 2026-02-27 17:21:05.442640: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4af0000 of size 327680 next 142 2026-02-27 17:21:05.442652: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4b40000 of size 786432 next 18446744073709551615 2026-02-27 17:21:05.442663: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 8388608 2026-02-27 17:21:05.442683: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f46f4c00000 of size 589824 next 160 2026-02-27 17:21:05.442695: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4c90000 of size 589824 next 158 2026-02-27 17:21:05.442707: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4d20000 of size 524288 next 171 2026-02-27 17:21:05.442718: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f46f4da0000 of size 1048576 next 222 2026-02-27 17:21:05.442730: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4ea0000 of size 1048576 next 190 2026-02-27 17:21:05.442741: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f4fa0000 of size 1048576 next 189 2026-02-27 17:21:05.442753: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f50a0000 of size 1048576 next 203 2026-02-27 17:21:05.442764: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f51a0000 of size 1048576 next 251 2026-02-27 17:21:05.442776: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f46f52a0000 of size 1441792 next 18446744073709551615 2026-02-27 17:21:05.442787: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 1048576 2026-02-27 17:21:05.442799: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e00000 of size 1280 next 1 2026-02-27 17:21:05.442811: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e00500 of size 256 next 5 2026-02-27 17:21:05.442822: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e00600 of size 256 next 8 2026-02-27 17:21:05.442834: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e00700 of size 256 next 9 2026-02-27 17:21:05.442845: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e00800 of size 256 next 10 2026-02-27 17:21:05.442857: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e00900 of size 256 next 11 2026-02-27 17:21:05.442868: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e00a00 of size 256 next 12 2026-02-27 17:21:05.442880: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e00b00 of size 256 next 13 2026-02-27 17:21:05.442891: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e00c00 of size 256 next 17 2026-02-27 17:21:05.442903: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e00d00 of size 256 next 19 2026-02-27 17:21:05.442914: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e00e00 of size 256 next 20 2026-02-27 17:21:05.442925: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e00f00 of size 256 next 21 2026-02-27 17:21:05.442937: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e01000 of size 256 next 22 2026-02-27 17:21:05.442948: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e01100 of size 256 next 24 2026-02-27 17:21:05.442960: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e01200 of size 256 next 25 2026-02-27 17:21:05.442971: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e01300 of size 256 next 23 2026-02-27 17:21:05.442983: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e01400 of size 256 next 28 2026-02-27 17:21:05.442994: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e01500 of size 256 next 29 2026-02-27 17:21:05.443006: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e01600 of size 256 next 30 2026-02-27 17:21:05.443017: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e01700 of size 256 next 31 2026-02-27 17:21:05.443028: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e01800 of size 256 next 33 2026-02-27 17:21:05.443040: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e01900 of size 256 next 34 2026-02-27 17:21:05.443059: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e01a00 of size 1024 next 32 2026-02-27 17:21:05.443071: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e01e00 of size 1024 next 37 2026-02-27 17:21:05.443083: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e02200 of size 1024 next 38 2026-02-27 17:21:05.443094: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e02600 of size 1024 next 39 2026-02-27 17:21:05.443106: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e02a00 of size 1024 next 40 2026-02-27 17:21:05.443117: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e02e00 of size 1024 next 41 2026-02-27 17:21:05.443129: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e03200 of size 1024 next 43 2026-02-27 17:21:05.443140: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e03600 of size 1024 next 44 2026-02-27 17:21:05.443152: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e03a00 of size 1024 next 45 2026-02-27 17:21:05.443163: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e03e00 of size 1024 next 46 2026-02-27 17:21:05.443175: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e04200 of size 256 next 48 2026-02-27 17:21:05.443186: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e04300 of size 256 next 49 2026-02-27 17:21:05.443198: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e04400 of size 256 next 50 2026-02-27 17:21:05.443209: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e04500 of size 256 next 51 2026-02-27 17:21:05.443221: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e04600 of size 256 next 52 2026-02-27 17:21:05.443232: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e04700 of size 256 next 53 2026-02-27 17:21:05.443244: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e04800 of size 256 next 56 2026-02-27 17:21:05.443255: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e04900 of size 256 next 57 2026-02-27 17:21:05.443267: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e04a00 of size 256 next 58 2026-02-27 17:21:05.443278: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e04b00 of size 256 next 14 2026-02-27 17:21:05.443289: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e04c00 of size 256 next 15 2026-02-27 17:21:05.443301: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e04d00 of size 256 next 16 2026-02-27 17:21:05.443312: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e04e00 of size 1024 next 60 2026-02-27 17:21:05.443324: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e05200 of size 1024 next 61 2026-02-27 17:21:05.443335: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e05600 of size 1024 next 62 2026-02-27 17:21:05.443347: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e05a00 of size 1024 next 63 2026-02-27 17:21:05.443359: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e05e00 of size 1024 next 64 2026-02-27 17:21:05.443370: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e06200 of size 256 next 66 2026-02-27 17:21:05.443382: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e06300 of size 256 next 67 2026-02-27 17:21:05.443393: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e06400 of size 256 next 68 2026-02-27 17:21:05.443405: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e06500 of size 256 next 69 2026-02-27 17:21:05.443416: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e06600 of size 256 next 70 2026-02-27 17:21:05.443427: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e06700 of size 256 next 73 2026-02-27 17:21:05.443446: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e06800 of size 256 next 74 2026-02-27 17:21:05.443458: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e06900 of size 256 next 75 2026-02-27 17:21:05.443470: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e06a00 of size 256 next 76 2026-02-27 17:21:05.443481: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e06b00 of size 256 next 77 2026-02-27 17:21:05.443493: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e06c00 of size 1024 next 79 2026-02-27 17:21:05.443504: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e07000 of size 1024 next 80 2026-02-27 17:21:05.443516: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e07400 of size 1024 next 81 2026-02-27 17:21:05.443527: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e07800 of size 1024 next 82 2026-02-27 17:21:05.443539: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e07c00 of size 1024 next 83 2026-02-27 17:21:05.443550: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e08000 of size 256 next 85 2026-02-27 17:21:05.443562: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e08100 of size 256 next 86 2026-02-27 17:21:05.443574: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e08200 of size 512 next 84 2026-02-27 17:21:05.443585: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e08400 of size 512 next 88 2026-02-27 17:21:05.443597: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e08600 of size 512 next 89 2026-02-27 17:21:05.443608: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e08800 of size 512 next 90 2026-02-27 17:21:05.443620: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e08a00 of size 512 next 91 2026-02-27 17:21:05.443631: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e08c00 of size 256 next 93 2026-02-27 17:21:05.443643: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e08d00 of size 256 next 94 2026-02-27 17:21:05.443654: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e08e00 of size 512 next 92 2026-02-27 17:21:05.443666: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e09000 of size 512 next 97 2026-02-27 17:21:05.443677: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e09200 of size 512 next 98 2026-02-27 17:21:05.443689: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e09400 of size 512 next 99 2026-02-27 17:21:05.443700: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e09600 of size 512 next 2 2026-02-27 17:21:05.443712: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e09800 of size 256 next 3 2026-02-27 17:21:05.443723: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e09900 of size 256 next 4 2026-02-27 17:21:05.443735: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e09a00 of size 16384 next 18 2026-02-27 17:21:05.443746: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e0da00 of size 256 next 100 2026-02-27 17:21:05.443758: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e0db00 of size 256 next 101 2026-02-27 17:21:05.443770: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e0dc00 of size 2048 next 104 2026-02-27 17:21:05.443781: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e0e400 of size 2048 next 105 2026-02-27 17:21:05.443793: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e0ec00 of size 2048 next 106 2026-02-27 17:21:05.443804: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e0f400 of size 2048 next 107 2026-02-27 17:21:05.443816: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e0fc00 of size 2048 next 108 2026-02-27 17:21:05.443835: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e10400 of size 256 next 109 2026-02-27 17:21:05.443847: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e10500 of size 256 next 110 2026-02-27 17:21:05.443858: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e10600 of size 2048 next 113 2026-02-27 17:21:05.443870: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e10e00 of size 2048 next 114 2026-02-27 17:21:05.443881: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e11600 of size 2048 next 115 2026-02-27 17:21:05.443892: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e11e00 of size 2048 next 116 2026-02-27 17:21:05.443904: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e12600 of size 2048 next 6 2026-02-27 17:21:05.443916: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e12e00 of size 37632 next 7 2026-02-27 17:21:05.443927: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e1c100 of size 512 next 118 2026-02-27 17:21:05.443939: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e1c300 of size 512 next 119 2026-02-27 17:21:05.443950: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e1c500 of size 512 next 120 2026-02-27 17:21:05.443962: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e1c700 of size 512 next 121 2026-02-27 17:21:05.443973: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e1c900 of size 512 next 122 2026-02-27 17:21:05.443985: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e1cb00 of size 512 next 125 2026-02-27 17:21:05.443996: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e1cd00 of size 512 next 126 2026-02-27 17:21:05.444008: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e1cf00 of size 512 next 127 2026-02-27 17:21:05.444019: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e1d100 of size 512 next 128 2026-02-27 17:21:05.444030: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e1d300 of size 512 next 129 2026-02-27 17:21:05.444042: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e1d500 of size 2048 next 130 2026-02-27 17:21:05.444053: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e1dd00 of size 2048 next 131 2026-02-27 17:21:05.444065: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e1e500 of size 2048 next 132 2026-02-27 17:21:05.444076: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e1ed00 of size 2048 next 133 2026-02-27 17:21:05.444088: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e1f500 of size 2048 next 134 2026-02-27 17:21:05.444099: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e1fd00 of size 512 next 136 2026-02-27 17:21:05.444111: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e1ff00 of size 512 next 137 2026-02-27 17:21:05.444122: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e20100 of size 512 next 138 2026-02-27 17:21:05.444134: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e20300 of size 512 next 139 2026-02-27 17:21:05.444145: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e20500 of size 512 next 140 2026-02-27 17:21:05.444157: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e20700 of size 512 next 141 2026-02-27 17:21:05.444168: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e20900 of size 512 next 143 2026-02-27 17:21:05.444179: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e20b00 of size 512 next 144 2026-02-27 17:21:05.444191: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e20d00 of size 512 next 145 2026-02-27 17:21:05.444210: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e20f00 of size 512 next 146 2026-02-27 17:21:05.444222: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e21100 of size 2048 next 147 2026-02-27 17:21:05.444233: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e21900 of size 2048 next 148 2026-02-27 17:21:05.444245: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e22100 of size 2048 next 149 2026-02-27 17:21:05.444256: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e22900 of size 2048 next 150 2026-02-27 17:21:05.444268: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e23100 of size 2048 next 151 2026-02-27 17:21:05.444279: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e23900 of size 512 next 153 2026-02-27 17:21:05.444291: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e23b00 of size 512 next 154 2026-02-27 17:21:05.444302: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e23d00 of size 512 next 155 2026-02-27 17:21:05.444314: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e23f00 of size 512 next 156 2026-02-27 17:21:05.444325: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e24100 of size 512 next 157 2026-02-27 17:21:05.444337: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e24300 of size 512 next 161 2026-02-27 17:21:05.444348: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e24500 of size 512 next 162 2026-02-27 17:21:05.444360: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e24700 of size 512 next 163 2026-02-27 17:21:05.444371: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e24900 of size 512 next 164 2026-02-27 17:21:05.444382: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e24b00 of size 512 next 165 2026-02-27 17:21:05.444394: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e24d00 of size 2048 next 166 2026-02-27 17:21:05.444405: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e25500 of size 2048 next 167 2026-02-27 17:21:05.444417: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e25d00 of size 2048 next 168 2026-02-27 17:21:05.444428: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e26500 of size 2048 next 169 2026-02-27 17:21:05.444440: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e26d00 of size 2048 next 170 2026-02-27 17:21:05.444451: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e27500 of size 1024 next 172 2026-02-27 17:21:05.444463: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e27900 of size 1024 next 173 2026-02-27 17:21:05.444474: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e27d00 of size 1024 next 174 2026-02-27 17:21:05.444486: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e28100 of size 1024 next 175 2026-02-27 17:21:05.444497: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e28500 of size 1024 next 176 2026-02-27 17:21:05.444508: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e28900 of size 256 next 178 2026-02-27 17:21:05.444520: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e28a00 of size 256 next 179 2026-02-27 17:21:05.444531: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e28b00 of size 1024 next 177 2026-02-27 17:21:05.444543: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e28f00 of size 1024 next 182 2026-02-27 17:21:05.444554: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e29300 of size 1024 next 183 2026-02-27 17:21:05.444566: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e29700 of size 1024 next 184 2026-02-27 17:21:05.444585: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e29b00 of size 1024 next 185 2026-02-27 17:21:05.444597: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e29f00 of size 256 next 187 2026-02-27 17:21:05.444608: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e2a000 of size 256 next 188 2026-02-27 17:21:05.444620: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e2a100 of size 4096 next 186 2026-02-27 17:21:05.444631: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e2b100 of size 4096 next 42 2026-02-27 17:21:05.444643: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e2c100 of size 65536 next 36 2026-02-27 17:21:05.444654: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e3c100 of size 65536 next 35 2026-02-27 17:21:05.444666: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e4c100 of size 4096 next 235 2026-02-27 17:21:05.444677: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e4d100 of size 4096 next 236 2026-02-27 17:21:05.444689: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e4e100 of size 4096 next 237 2026-02-27 17:21:05.444700: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e4f100 of size 4096 next 238 2026-02-27 17:21:05.444712: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e50100 of size 1024 next 239 2026-02-27 17:21:05.444723: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e50500 of size 1024 next 240 2026-02-27 17:21:05.444735: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e50900 of size 1024 next 241 2026-02-27 17:21:05.444746: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e50d00 of size 1024 next 242 2026-02-27 17:21:05.444757: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e51100 of size 1024 next 243 2026-02-27 17:21:05.444769: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e51500 of size 1024 next 246 2026-02-27 17:21:05.444780: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e51900 of size 1024 next 247 2026-02-27 17:21:05.444792: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e51d00 of size 1024 next 248 2026-02-27 17:21:05.444803: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e52100 of size 1024 next 249 2026-02-27 17:21:05.444815: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e52500 of size 1024 next 250 2026-02-27 17:21:05.444826: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e52900 of size 4096 next 252 2026-02-27 17:21:05.444838: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e53900 of size 4096 next 253 2026-02-27 17:21:05.444849: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e54900 of size 4096 next 254 2026-02-27 17:21:05.444861: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e55900 of size 4096 next 255 2026-02-27 17:21:05.444872: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e56900 of size 4096 next 256 2026-02-27 17:21:05.444884: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e57900 of size 1024 next 257 2026-02-27 17:21:05.444895: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e57d00 of size 1024 next 258 2026-02-27 17:21:05.444907: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e58100 of size 1024 next 259 2026-02-27 17:21:05.444918: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e58500 of size 1024 next 260 2026-02-27 17:21:05.444929: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e58900 of size 1024 next 261 2026-02-27 17:21:05.444941: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e58d00 of size 1024 next 264 2026-02-27 17:21:05.444952: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e59100 of size 1024 next 265 2026-02-27 17:21:05.444971: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e59500 of size 1024 next 266 2026-02-27 17:21:05.444983: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e59900 of size 1024 next 267 2026-02-27 17:21:05.444995: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e59d00 of size 1024 next 268 2026-02-27 17:21:05.445006: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e5a100 of size 4096 next 270 2026-02-27 17:21:05.445018: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e5b100 of size 4096 next 271 2026-02-27 17:21:05.445029: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e5c100 of size 4096 next 272 2026-02-27 17:21:05.445041: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e5d100 of size 4096 next 273 2026-02-27 17:21:05.445052: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e5e100 of size 4096 next 274 2026-02-27 17:21:05.445063: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e5f100 of size 1024 next 275 2026-02-27 17:21:05.445075: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e5f500 of size 1024 next 276 2026-02-27 17:21:05.445086: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e5f900 of size 1024 next 277 2026-02-27 17:21:05.445098: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e5fd00 of size 1024 next 278 2026-02-27 17:21:05.445109: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e60100 of size 1024 next 279 2026-02-27 17:21:05.445121: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e60500 of size 1024 next 282 2026-02-27 17:21:05.445132: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e60900 of size 1024 next 283 2026-02-27 17:21:05.445144: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e60d00 of size 1024 next 284 2026-02-27 17:21:05.445155: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e61100 of size 1024 next 285 2026-02-27 17:21:05.445167: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e61500 of size 1024 next 286 2026-02-27 17:21:05.445178: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e61900 of size 4096 next 288 2026-02-27 17:21:05.445190: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e62900 of size 6144 next 27 2026-02-27 17:21:05.445202: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e64100 of size 147456 next 26 2026-02-27 17:21:05.445213: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e88100 of size 65536 next 47 2026-02-27 17:21:05.445225: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830e98100 of size 65536 next 59 2026-02-27 17:21:05.445236: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830ea8100 of size 65536 next 65 2026-02-27 17:21:05.445248: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f4830eb8100 of size 294656 next 18446744073709551615 2026-02-27 17:21:05.445259: I tensorflow/core/common_runtime/bfc_allocator.cc:995] Summary of in-use Chunks by size: 2026-02-27 17:21:05.445274: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 60 Chunks of size 256 totalling 15.0KiB 2026-02-27 17:21:05.445287: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 40 Chunks of size 512 totalling 20.0KiB 2026-02-27 17:21:05.445300: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 135 Chunks of size 1024 totalling 135.0KiB 2026-02-27 17:21:05.445313: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1280 totalling 1.2KiB 2026-02-27 17:21:05.445326: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 25 Chunks of size 2048 totalling 50.0KiB 2026-02-27 17:21:05.445338: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 58 Chunks of size 4096 totalling 232.0KiB 2026-02-27 17:21:05.445359: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 6144 totalling 6.0KiB 2026-02-27 17:21:05.445372: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 7680 totalling 7.5KiB 2026-02-27 17:21:05.445385: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 16384 totalling 16.0KiB 2026-02-27 17:21:05.445397: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 37632 totalling 36.8KiB 2026-02-27 17:21:05.445410: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 6 Chunks of size 65536 totalling 384.0KiB 2026-02-27 17:21:05.445423: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 131072 totalling 128.0KiB 2026-02-27 17:21:05.445435: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 147456 totalling 432.0KiB 2026-02-27 17:21:05.445448: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 4 Chunks of size 262144 totalling 1.00MiB 2026-02-27 17:21:05.445461: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 294656 totalling 287.8KiB 2026-02-27 17:21:05.445473: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 327680 totalling 640.0KiB 2026-02-27 17:21:05.445486: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 524288 totalling 1.00MiB 2026-02-27 17:21:05.445498: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 589824 totalling 1.69MiB 2026-02-27 17:21:05.445511: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 786432 totalling 768.0KiB 2026-02-27 17:21:05.445524: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 13 Chunks of size 1048576 totalling 13.00MiB 2026-02-27 17:21:05.445536: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 7 Chunks of size 1310720 totalling 8.75MiB 2026-02-27 17:21:05.445549: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1441792 totalling 1.38MiB 2026-02-27 17:21:05.445561: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1835008 totalling 1.75MiB 2026-02-27 17:21:05.445574: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 2097152 totalling 2.00MiB 2026-02-27 17:21:05.445587: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 10 Chunks of size 2359296 totalling 22.50MiB 2026-02-27 17:21:05.445599: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 2883584 totalling 2.75MiB 2026-02-27 17:21:05.445612: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 3903488 totalling 3.72MiB 2026-02-27 17:21:05.445624: I tensorflow/core/common_runtime/bfc_allocator.cc:1002] Sum Total of in-use chunks: 62.62MiB 2026-02-27 17:21:05.445636: I tensorflow/core/common_runtime/bfc_allocator.cc:1004] total_region_allocated_bytes_: 69963776 memory_limit_: 195756032 available bytes: 125792256 curr_region_allocation_bytes_: 134217728 2026-02-27 17:21:05.445652: I tensorflow/core/common_runtime/bfc_allocator.cc:1010] Stats: Limit: 195756032 InUse: 65662208 MaxInUse: 65662208 NumAllocs: 1266 MaxAllocSize: 4325376 2026-02-27 17:21:05.445675: W tensorflow/core/common_runtime/bfc_allocator.cc:439] *****_**********************************************************************x************_********** 2026-02-27 17:21:05.445719: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at cwise_ops_common.h:134 : Resource exhausted: OOM when allocating tensor with shape[3,3,256,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2847 thcls : [{'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}] thcl {'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5275 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5275, 'learn_RUBBIA_REFUS_AMIENS_23', 16384, 25088, 'learn_RUBBIA_REFUS_AMIENS_23', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 23, 14, 19, 39), datetime.datetime(2021, 4, 23, 14, 19, 39)) {'thcl': {'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'], 'list_hashtags_csv': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'svm_hashtag_type_desc': 5275, 'photo_desc_type': 5275, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_RUBBIA_REFUS_AMIENS_23 NUM_CLASSES 9 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 Exception in mask_detect : OOM when allocating tensor with shape[3,3,256,256] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Mul] we want to redo the detection Using TensorFlow backend. max_time_sub_proc : 3600 erreur pendant la detection Catched exception ! (2013, 'Lost connection to MySQL server during query') Connect or reconnect ! in case -12 caffe_path_current : About to save ! 2 After save, about to update current ! 3.95user 3.16system 0:39.43elapsed 18%CPU (0avgtext+0avgdata 1114168maxresident)k 104inputs+22488outputs (2major+241720minor)pagefaults 0swaps