python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' -a 4741 -l 30 ' -s datou_current_4741 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 741267 load datou : 4741 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 13156 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 13160 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 13160 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 13157 crop_condition is not consistent : 4 used against 2 in the step definition ! WARNING : number of outputs for step 13157 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 13161 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 13159 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 13159 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 13156 doesn't seem to be define in the database( WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 13161 doesn't seem to be define in the database( WARNING : type of input 3 of step 13159 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13163 have datatype=6 whereas input 2 of step 13157 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13162 have datatype=6 whereas input 2 of step 13157 have datatype=None We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 13161 have datatype=10 whereas input 3 of step 13164 have datatype=6 WARNING : output 0 of step 13161 have datatype=10 whereas input 0 of step 13166 have datatype=18 WARNING : type of input 5 of step 13164 doesn't seem to be define in the database( WARNING : output 0 of step 13166 have datatype=11 whereas input 5 of step 13164 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 1 of step 13158 have datatype=7 whereas input 2 of step 13157 have datatype=None WARNING : type of output 3 of step 13157 doesn't seem to be define in the database( WARNING : type of input 3 of step 13161 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13163 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13162 doesn't seem to be define in the database( DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5732, 'mask_refus_amiens_050123', 16384, 25088, 'mask_refus_amiens_050123', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 1, 5, 13, 23, 59), datetime.datetime(2023, 1, 5, 13, 23, 59)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5972, 'learn_entrant_syctomXV_111023', 16384, 25088, 'learn_entrant_syctomXV_111023', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 10, 11, 15, 57, 35), datetime.datetime(2023, 10, 11, 15, 57, 35)) load thcls load THCL from format json or kwargs add thcl : 3453 in CacheModelConfig load THCL from format json or kwargs add thcl : 3783 in CacheModelConfig load pdts add pdt : 5732 in CacheModelConfig add pdt : 5972 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 list_input_json: [] Current got : datou_id : 4741, datou_cur_ids : ['3628598'] with mtr_portfolio_ids : ['26307204'] and first list_photo_ids : [] new path : /proc/741267/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 13156 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 13160 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 13160 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 13157 crop_condition is not consistent : 4 used against 2 in the step definition ! WARNING : number of outputs for step 13157 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 13161 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 13159 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 13159 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 13156 doesn't seem to be define in the database( WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 13161 doesn't seem to be define in the database( WARNING : type of input 3 of step 13159 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13163 have datatype=6 whereas input 2 of step 13157 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13162 have datatype=6 whereas input 2 of step 13157 have datatype=None We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 13161 have datatype=10 whereas input 3 of step 13164 have datatype=6 WARNING : output 0 of step 13161 have datatype=10 whereas input 0 of step 13166 have datatype=18 WARNING : type of input 5 of step 13164 doesn't seem to be define in the database( WARNING : output 0 of step 13166 have datatype=11 whereas input 5 of step 13164 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 1 of step 13158 have datatype=7 whereas input 2 of step 13157 have datatype=None WARNING : type of output 3 of step 13157 doesn't seem to be define in the database( WARNING : type of input 3 of step 13161 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13163 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13162 doesn't seem to be define in the database( DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, mask_detect, blur_detection, brightness, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , BFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 4 ; length of list_pids : 4 ; length of list_args : 4 time to download the photos : 0.7748746871948242 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Thu Aug 28 22:21:28 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5628 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-08-28 22:21:31.072235: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-08-28 22:21:31.100647: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3492910000 Hz 2025-08-28 22:21:31.102247: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f12d8000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-08-28 22:21:31.102275: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-08-28 22:21:31.105467: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-08-28 22:21:31.257410: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1cc96980 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-08-28 22:21:31.257465: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-08-28 22:21:31.258279: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-08-28 22:21:31.258761: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-08-28 22:21:31.262390: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-08-28 22:21:31.266643: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-08-28 22:21:31.267041: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-08-28 22:21:31.269513: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-08-28 22:21:31.270594: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-08-28 22:21:31.275822: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-08-28 22:21:31.277079: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-08-28 22:21:31.277168: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-08-28 22:21:31.277816: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-08-28 22:21:31.277832: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-08-28 22:21:31.277841: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-08-28 22:21:31.278863: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5147 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-08-28 22:21:31.687603: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-08-28 22:21:31.687771: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-08-28 22:21:31.688217: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-08-28 22:21:31.688241: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-08-28 22:21:31.688257: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-08-28 22:21:31.688273: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-08-28 22:21:31.688290: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-08-28 22:21:31.688308: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-08-28 22:21:31.689319: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-08-28 22:21:31.690655: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-08-28 22:21:31.690708: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-08-28 22:21:31.690725: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-08-28 22:21:31.690739: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-08-28 22:21:31.690774: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-08-28 22:21:31.690789: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-08-28 22:21:31.690803: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-08-28 22:21:31.690817: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-08-28 22:21:31.691778: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-08-28 22:21:31.691828: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-08-28 22:21:31.691838: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-08-28 22:21:31.691845: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-08-28 22:21:31.692838: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5147 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl3783 thcls : [{'id': 3783, 'mtr_user_id': 31, 'name': 'learn_entrant_syctomXV_111023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'photo_desc_type': 5972, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 3783, 'mtr_user_id': 31, 'name': 'learn_entrant_syctomXV_111023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'photo_desc_type': 5972, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5972 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5972, 'learn_entrant_syctomXV_111023', 16384, 25088, 'learn_entrant_syctomXV_111023', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 10, 11, 15, 57, 35), datetime.datetime(2023, 10, 11, 15, 57, 35)) {'thcl': {'id': 3783, 'mtr_user_id': 31, 'name': 'learn_entrant_syctomXV_111023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'photo_desc_type': 5972, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'sac', 'textiles', 'verre', 'organique', 'dasri', 'masque', 'encombrant', 'autre_emballage', 'autre_non_emballage', 'environnement'], 'list_hashtags_csv': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'svm_hashtag_type_desc': 5972, 'photo_desc_type': 5972, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'sac', 'textiles', 'verre', 'organique', 'dasri', 'masque', 'encombrant', 'autre_emballage', 'autre_non_emballage', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_entrant_syctomXV_111023 NUM_CLASSES 11 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_entrant_syctomXV_111023 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-08-28 22:21:41.830788: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-08-28 22:21:42.068692: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-08-28 22:21:44.337676: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:44.338368: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.60G (3865470464 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.231000: W tensorflow/core/common_runtime/bfc_allocator.cc:311] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature. 2025-08-28 22:21:45.315031: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.315391: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-08-28 22:21:45.316121: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.316143: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-08-28 22:21:45.322591: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.322614: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.78GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-08-28 22:21:45.323238: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.323253: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.78GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-08-28 22:21:45.348690: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.348805: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 19.91MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-08-28 22:21:45.348826: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-08-28 22:21:45.350035: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.350066: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 16.00MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-08-28 22:21:45.351240: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.351270: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 16.00MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-08-28 22:21:45.358498: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.358553: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 63.85MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-08-28 22:21:45.359225: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.359242: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 63.85MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-08-28 22:21:45.359934: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.359951: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.26GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-08-28 22:21:45.360621: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.368863: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.369569: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.386141: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.387180: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.388167: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.389155: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.393963: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.394979: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.395975: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.396970: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.398973: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.409542: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.410838: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.422176: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.423296: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.424358: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.425393: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.426439: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-08-28 22:21:45.427479: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/learn_entrant_syctomXV_111023 /data/models_weight/learn_entrant_syctomXV_111023/mask_model.h5 size_local : 256052544 size in s3 : 256052544 create time local : 2023-10-26 16:36:43 create time in s3 : 2023-10-11 13:57:17 mask_model.h5 already exist and didn't need to update list_images length : 4 NEW PHOTO Processing 1 images image shape: (2448, 2448, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 19) min: 0.00000 max: 2448.00000 nb d'objets trouves : 18 NEW PHOTO Processing 1 images image shape: (2448, 2448, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 19) min: 0.00000 max: 2448.00000 nb d'objets trouves : 9 NEW PHOTO Processing 1 images image shape: (2448, 2448, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 19) min: 0.00000 max: 2448.00000 nb d'objets trouves : 20 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -122.88750 max: 151.10000 image_metas shape: (1, 19) min: 0.00000 max: 3120.00000 nb d'objets trouves : 6 Detection mask done ! Trying to reset tf kernel 741457 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 3636 tf kernel not reseted sub process len(results) : 4 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 4 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 4435 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl3783 Catched exception ! Connect or reconnect ! thcls : [{'id': 3783, 'mtr_user_id': 31, 'name': 'learn_entrant_syctomXV_111023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'photo_desc_type': 5972, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 3783, 'mtr_user_id': 31, 'name': 'learn_entrant_syctomXV_111023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'photo_desc_type': 5972, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5972 ['background', 'sac', 'textiles', 'verre', 'organique', 'dasri', 'masque', 'encombrant', 'autre_emballage', 'autre_non_emballage', 'environnement'] time for calcul the mask position with numpy : 0.004442453384399414 nb_pixel_total : 107966 time to create 1 rle with old method : 0.13326001167297363 length of segment : 310 time for calcul the mask position with numpy : 0.001706838607788086 nb_pixel_total : 62868 time to create 1 rle with old method : 0.07104134559631348 length of segment : 248 time for calcul the mask position with numpy : 0.0007600784301757812 nb_pixel_total : 27110 time to create 1 rle with old method : 0.0305635929107666 length of segment : 241 time for calcul the mask position with numpy : 0.0002143383026123047 nb_pixel_total : 6699 time to create 1 rle with old method : 0.007731199264526367 length of segment : 149 time for calcul the mask position with numpy : 0.0006875991821289062 nb_pixel_total : 20913 time to create 1 rle with old method : 0.023211002349853516 length of segment : 163 time for calcul the mask position with numpy : 0.004362583160400391 nb_pixel_total : 159410 time to create 1 rle with new method : 0.012949705123901367 length of segment : 311 time for calcul the mask position with numpy : 0.0004451274871826172 nb_pixel_total : 10501 time to create 1 rle with old method : 0.01732778549194336 length of segment : 171 time for calcul the mask position with numpy : 0.0005543231964111328 nb_pixel_total : 10782 time to create 1 rle with old method : 0.017962932586669922 length of segment : 145 time for calcul the mask position with numpy : 0.001074075698852539 nb_pixel_total : 22610 time to create 1 rle with old method : 0.0435032844543457 length of segment : 184 time for calcul the mask position with numpy : 0.0008983612060546875 nb_pixel_total : 21317 time to create 1 rle with old method : 0.0403287410736084 length of segment : 161 time for calcul the mask position with numpy : 0.0004892349243164062 nb_pixel_total : 7027 time to create 1 rle with old method : 0.014106035232543945 length of segment : 116 time for calcul the mask position with numpy : 0.0005867481231689453 nb_pixel_total : 9990 time to create 1 rle with old method : 0.019057512283325195 length of segment : 131 time for calcul the mask position with numpy : 0.00021457672119140625 nb_pixel_total : 6219 time to create 1 rle with old method : 0.01192021369934082 length of segment : 75 time for calcul the mask position with numpy : 0.0007996559143066406 nb_pixel_total : 24587 time to create 1 rle with old method : 0.03803277015686035 length of segment : 194 time for calcul the mask position with numpy : 0.00021529197692871094 nb_pixel_total : 3191 time to create 1 rle with old method : 0.0042972564697265625 length of segment : 61 time for calcul the mask position with numpy : 0.00018835067749023438 nb_pixel_total : 4335 time to create 1 rle with old method : 0.005246162414550781 length of segment : 84 time for calcul the mask position with numpy : 0.0005249977111816406 nb_pixel_total : 23178 time to create 1 rle with old method : 0.026592493057250977 length of segment : 186 time for calcul the mask position with numpy : 0.00020837783813476562 nb_pixel_total : 4820 time to create 1 rle with old method : 0.007113933563232422 length of segment : 98 time for calcul the mask position with numpy : 0.061057329177856445 nb_pixel_total : 401552 time to create 1 rle with new method : 0.14237618446350098 length of segment : 723 time for calcul the mask position with numpy : 0.00047898292541503906 nb_pixel_total : 14004 time to create 1 rle with old method : 0.016318798065185547 length of segment : 170 time for calcul the mask position with numpy : 0.0007915496826171875 nb_pixel_total : 19952 time to create 1 rle with old method : 0.02385997772216797 length of segment : 210 time for calcul the mask position with numpy : 0.00030684471130371094 nb_pixel_total : 9007 time to create 1 rle with old method : 0.010484695434570312 length of segment : 121 time for calcul the mask position with numpy : 0.0004837512969970703 nb_pixel_total : 18113 time to create 1 rle with old method : 0.021549463272094727 length of segment : 143 time for calcul the mask position with numpy : 0.0006687641143798828 nb_pixel_total : 22697 time to create 1 rle with old method : 0.026421070098876953 length of segment : 222 time for calcul the mask position with numpy : 0.0020554065704345703 nb_pixel_total : 123701 time to create 1 rle with old method : 0.13831877708435059 length of segment : 464 time for calcul the mask position with numpy : 0.0007107257843017578 nb_pixel_total : 5976 time to create 1 rle with old method : 0.00738072395324707 length of segment : 164 time for calcul the mask position with numpy : 0.0005223751068115234 nb_pixel_total : 6677 time to create 1 rle with old method : 0.00853872299194336 length of segment : 86 time for calcul the mask position with numpy : 0.003185272216796875 nb_pixel_total : 142123 time to create 1 rle with old method : 0.1617577075958252 length of segment : 369 time for calcul the mask position with numpy : 0.0007686614990234375 nb_pixel_total : 25963 time to create 1 rle with old method : 0.031221866607666016 length of segment : 257 time for calcul the mask position with numpy : 0.0005762577056884766 nb_pixel_total : 16881 time to create 1 rle with old method : 0.02192974090576172 length of segment : 126 time for calcul the mask position with numpy : 0.0005853176116943359 nb_pixel_total : 19388 time to create 1 rle with old method : 0.024346351623535156 length of segment : 240 time for calcul the mask position with numpy : 0.013216733932495117 nb_pixel_total : 390240 time to create 1 rle with new method : 0.40227174758911133 length of segment : 506 time for calcul the mask position with numpy : 0.00025725364685058594 nb_pixel_total : 10053 time to create 1 rle with old method : 0.011886835098266602 length of segment : 102 time for calcul the mask position with numpy : 0.0011773109436035156 nb_pixel_total : 41097 time to create 1 rle with old method : 0.04925179481506348 length of segment : 147 time for calcul the mask position with numpy : 0.00022983551025390625 nb_pixel_total : 4839 time to create 1 rle with old method : 0.005883455276489258 length of segment : 105 time for calcul the mask position with numpy : 0.0003905296325683594 nb_pixel_total : 16025 time to create 1 rle with old method : 0.019267559051513672 length of segment : 149 time for calcul the mask position with numpy : 0.00023555755615234375 nb_pixel_total : 9239 time to create 1 rle with old method : 0.010977745056152344 length of segment : 123 time for calcul the mask position with numpy : 0.00021147727966308594 nb_pixel_total : 6647 time to create 1 rle with old method : 0.008305788040161133 length of segment : 72 time for calcul the mask position with numpy : 0.00016808509826660156 nb_pixel_total : 3355 time to create 1 rle with old method : 0.004157304763793945 length of segment : 157 time for calcul the mask position with numpy : 0.0004622936248779297 nb_pixel_total : 12673 time to create 1 rle with old method : 0.016289234161376953 length of segment : 70 time for calcul the mask position with numpy : 0.0019311904907226562 nb_pixel_total : 69555 time to create 1 rle with old method : 0.0844414234161377 length of segment : 413 time for calcul the mask position with numpy : 0.0003287792205810547 nb_pixel_total : 12605 time to create 1 rle with old method : 0.015122652053833008 length of segment : 109 time for calcul the mask position with numpy : 0.000579833984375 nb_pixel_total : 11291 time to create 1 rle with old method : 0.013300180435180664 length of segment : 156 time for calcul the mask position with numpy : 0.0006456375122070312 nb_pixel_total : 15736 time to create 1 rle with old method : 0.020232439041137695 length of segment : 142 time for calcul the mask position with numpy : 0.0007979869842529297 nb_pixel_total : 14625 time to create 1 rle with old method : 0.01696038246154785 length of segment : 309 time for calcul the mask position with numpy : 0.0007040500640869141 nb_pixel_total : 16813 time to create 1 rle with old method : 0.019816160202026367 length of segment : 164 time for calcul the mask position with numpy : 0.00015997886657714844 nb_pixel_total : 4569 time to create 1 rle with old method : 0.00566554069519043 length of segment : 117 time for calcul the mask position with numpy : 0.002391338348388672 nb_pixel_total : 139071 time to create 1 rle with old method : 0.16105031967163086 length of segment : 604 time for calcul the mask position with numpy : 0.0008592605590820312 nb_pixel_total : 36849 time to create 1 rle with old method : 0.04317283630371094 length of segment : 271 time for calcul the mask position with numpy : 0.0005140304565429688 nb_pixel_total : 30642 time to create 1 rle with old method : 0.03540301322937012 length of segment : 203 time for calcul the mask position with numpy : 0.8214797973632812 nb_pixel_total : 4256319 time to create 1 rle with new method : 1.0612046718597412 length of segment : 2699 time for calcul the mask position with numpy : 0.48023366928100586 nb_pixel_total : 3216702 time to create 1 rle with new method : 0.32723069190979004 length of segment : 2317 time for calcul the mask position with numpy : 0.2165067195892334 nb_pixel_total : 4716102 time to create 1 rle with new method : 0.9352266788482666 length of segment : 3917 time spent for convertir_results : 12.501463174819946 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 53 chid ids of type : 4854 Number RLEs to save : 0 save missing photos in datou_result : time spend for datou_step_exec : 43.5495867729187 time spend to save output : 0.10796976089477539 total time spend for step 1 : 43.65755653381348 step2:mask_detect Thu Aug 28 22:22:11 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3000 l 3632 free memory gpu now : 567 wait 20 seconds l 3637 free memory gpu now : 567 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-08-28 22:22:35.930982: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-08-28 22:22:35.964598: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3492910000 Hz 2025-08-28 22:22:35.967060: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f12d4000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-08-28 22:22:35.967095: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-08-28 22:22:35.972204: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-08-28 22:22:36.166867: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1d7b0ac0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-08-28 22:22:36.166961: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-08-28 22:22:36.168222: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-08-28 22:22:36.170408: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-08-28 22:22:36.199966: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-08-28 22:22:36.220181: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-08-28 22:22:36.224999: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-08-28 22:22:36.263778: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-08-28 22:22:36.268652: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-08-28 22:22:36.319828: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-08-28 22:22:36.321282: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-08-28 22:22:36.321806: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-08-28 22:22:36.322953: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-08-28 22:22:36.322981: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-08-28 22:22:36.322996: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-08-28 22:22:36.325046: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 190 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-08-28 22:22:36.500778: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-08-28 22:22:36.500890: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-08-28 22:22:36.500916: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-08-28 22:22:36.500939: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-08-28 22:22:36.500961: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-08-28 22:22:36.500983: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-08-28 22:22:36.501005: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-08-28 22:22:36.501046: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-08-28 22:22:36.501974: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-08-28 22:22:36.503195: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-08-28 22:22:36.503310: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-08-28 22:22:36.503342: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-08-28 22:22:36.503370: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-08-28 22:22:36.503395: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-08-28 22:22:36.503423: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-08-28 22:22:36.503449: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-08-28 22:22:36.503478: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-08-28 22:22:36.504428: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-08-28 22:22:36.504470: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-08-28 22:22:36.504482: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-08-28 22:22:36.504492: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-08-28 22:22:36.505403: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 190 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) 2025-08-28 22:22:49.433848: W tensorflow/core/common_runtime/bfc_allocator.cc:434] Allocator (GPU_0_bfc) ran out of memory trying to allocate 9.00MiB (rounded to 9437184) Current allocation summary follows. 2025-08-28 22:22:49.433914: I tensorflow/core/common_runtime/bfc_allocator.cc:934] BFCAllocator dump for GPU_0_bfc 2025-08-28 22:22:49.433927: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (256): Total Chunks: 65, Chunks in use: 65. 16.2KiB allocated for chunks. 16.2KiB in use in bin. 8.9KiB client-requested in use in bin. 2025-08-28 22:22:49.433936: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (512): Total Chunks: 40, Chunks in use: 40. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin. 2025-08-28 22:22:49.433946: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1024): Total Chunks: 251, Chunks in use: 251. 251.2KiB allocated for chunks. 251.2KiB in use in bin. 251.0KiB client-requested in use in bin. 2025-08-28 22:22:49.433955: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2048): Total Chunks: 40, Chunks in use: 40. 80.0KiB allocated for chunks. 80.0KiB in use in bin. 80.0KiB client-requested in use in bin. 2025-08-28 22:22:49.433964: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4096): Total Chunks: 120, Chunks in use: 120. 487.5KiB allocated for chunks. 487.5KiB in use in bin. 480.0KiB client-requested in use in bin. 2025-08-28 22:22:49.433973: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8192): Total Chunks: 10, Chunks in use: 10. 80.0KiB allocated for chunks. 80.0KiB in use in bin. 80.0KiB client-requested in use in bin. 2025-08-28 22:22:49.433982: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16384): Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin. 2025-08-28 22:22:49.433991: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (32768): Total Chunks: 1, Chunks in use: 1. 36.8KiB allocated for chunks. 36.8KiB in use in bin. 36.8KiB client-requested in use in bin. 2025-08-28 22:22:49.434001: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (65536): Total Chunks: 6, Chunks in use: 6. 384.0KiB allocated for chunks. 384.0KiB in use in bin. 384.0KiB client-requested in use in bin. 2025-08-28 22:22:49.434009: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (131072): Total Chunks: 4, Chunks in use: 4. 560.0KiB allocated for chunks. 560.0KiB in use in bin. 560.0KiB client-requested in use in bin. 2025-08-28 22:22:49.434018: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (262144): Total Chunks: 8, Chunks in use: 7. 2.35MiB allocated for chunks. 1.91MiB in use in bin. 1.75MiB client-requested in use in bin. 2025-08-28 22:22:49.434027: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (524288): Total Chunks: 7, Chunks in use: 6. 4.39MiB allocated for chunks. 3.44MiB in use in bin. 3.25MiB client-requested in use in bin. 2025-08-28 22:22:49.434036: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1048576): Total Chunks: 48, Chunks in use: 45. 54.12MiB allocated for chunks. 50.62MiB in use in bin. 45.00MiB client-requested in use in bin. 2025-08-28 22:22:49.434065: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2097152): Total Chunks: 28, Chunks in use: 25. 64.25MiB allocated for chunks. 55.75MiB in use in bin. 55.75MiB client-requested in use in bin. 2025-08-28 22:22:49.434074: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4194304): Total Chunks: 4, Chunks in use: 2. 18.00MiB allocated for chunks. 8.00MiB in use in bin. 8.00MiB client-requested in use in bin. 2025-08-28 22:22:49.434083: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8388608): Total Chunks: 5, Chunks in use: 4. 45.44MiB allocated for chunks. 37.44MiB in use in bin. 35.00MiB client-requested in use in bin. 2025-08-28 22:22:49.434092: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-08-28 22:22:49.434099: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-08-28 22:22:49.434107: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-08-28 22:22:49.434115: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-08-28 22:22:49.434123: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-08-28 22:22:49.434131: I tensorflow/core/common_runtime/bfc_allocator.cc:957] Bin for 9.00MiB was 8.00MiB, Chunk State: 2025-08-28 22:22:49.434146: I tensorflow/core/common_runtime/bfc_allocator.cc:963] Size: 8.00MiB | Requested Size: 8.00MiB | in_use: 0 | bin_num: 15, prev: Size: 9.00MiB | Requested Size: 9.00MiB | in_use: 1 | bin_num: -1, next: Size: 8.00MiB | Requested Size: 8.00MiB | in_use: 1 | bin_num: -1 2025-08-28 22:22:49.434153: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 66519040 2025-08-28 22:22:49.434162: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f10f6000000 of size 4194304 next 632 2025-08-28 22:22:49.434171: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10f6400000 of size 4194304 next 617 2025-08-28 22:22:49.434178: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10f6800000 of size 4194304 next 616 2025-08-28 22:22:49.434185: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f10f6c00000 of size 6291456 next 608 2025-08-28 22:22:49.434192: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10f7200000 of size 9437184 next 607 2025-08-28 22:22:49.434199: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f10f7b00000 of size 8388608 next 625 2025-08-28 22:22:49.434207: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10f8300000 of size 8388608 next 624 2025-08-28 22:22:49.434214: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10f8b00000 of size 9437184 next 637 2025-08-28 22:22:49.434221: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10f9400000 of size 11993088 next 18446744073709551615 2025-08-28 22:22:49.434228: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 67108864 2025-08-28 22:22:49.434235: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fa000000 of size 2359296 next 370 2025-08-28 22:22:49.434242: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fa240000 of size 1048576 next 394 2025-08-28 22:22:49.434249: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f10fa340000 of size 1310720 next 388 2025-08-28 22:22:49.434256: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fa480000 of size 2359296 next 387 2025-08-28 22:22:49.434269: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fa6c0000 of size 1048576 next 412 2025-08-28 22:22:49.434276: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fa7c0000 of size 1310720 next 406 2025-08-28 22:22:49.434283: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fa900000 of size 2359296 next 405 2025-08-28 22:22:49.434290: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fab40000 of size 1048576 next 430 2025-08-28 22:22:49.434297: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fac40000 of size 1310720 next 424 2025-08-28 22:22:49.434304: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fad80000 of size 2359296 next 423 2025-08-28 22:22:49.434311: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fafc0000 of size 1048576 next 447 2025-08-28 22:22:49.434318: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fb0c0000 of size 1310720 next 441 2025-08-28 22:22:49.434325: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fb200000 of size 2359296 next 440 2025-08-28 22:22:49.434332: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fb440000 of size 1048576 next 465 2025-08-28 22:22:49.434339: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fb540000 of size 1310720 next 459 2025-08-28 22:22:49.434345: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fb680000 of size 2359296 next 458 2025-08-28 22:22:49.434352: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fb8c0000 of size 1048576 next 483 2025-08-28 22:22:49.434359: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fb9c0000 of size 1310720 next 477 2025-08-28 22:22:49.434366: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fbb00000 of size 2359296 next 476 2025-08-28 22:22:49.434373: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fbd40000 of size 1048576 next 501 2025-08-28 22:22:49.434380: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fbe40000 of size 1310720 next 495 2025-08-28 22:22:49.434387: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fbf80000 of size 2359296 next 494 2025-08-28 22:22:49.434394: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fc1c0000 of size 1048576 next 519 2025-08-28 22:22:49.434401: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fc2c0000 of size 1310720 next 513 2025-08-28 22:22:49.434407: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fc400000 of size 2359296 next 512 2025-08-28 22:22:49.434414: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fc640000 of size 1048576 next 537 2025-08-28 22:22:49.434421: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fc740000 of size 1310720 next 531 2025-08-28 22:22:49.434428: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fc880000 of size 2359296 next 530 2025-08-28 22:22:49.434435: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fcac0000 of size 1048576 next 555 2025-08-28 22:22:49.434442: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fcbc0000 of size 1310720 next 549 2025-08-28 22:22:49.434449: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fcd00000 of size 2359296 next 548 2025-08-28 22:22:49.434456: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fcf40000 of size 1048576 next 573 2025-08-28 22:22:49.434463: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fd040000 of size 1310720 next 567 2025-08-28 22:22:49.434470: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fd180000 of size 2359296 next 566 2025-08-28 22:22:49.434477: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fd3c0000 of size 1048576 next 591 2025-08-28 22:22:49.434489: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f10fd4c0000 of size 1310720 next 585 2025-08-28 22:22:49.434496: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fd600000 of size 2359296 next 584 2025-08-28 22:22:49.434503: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f10fd840000 of size 2097152 next 597 2025-08-28 22:22:49.434511: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fda40000 of size 2097152 next 596 2025-08-28 22:22:49.434518: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f10fdc40000 of size 3932160 next 18446744073709551615 2025-08-28 22:22:49.434525: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 33554432 2025-08-28 22:22:49.434532: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fe000000 of size 2359296 next 245 2025-08-28 22:22:49.434539: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fe240000 of size 1048576 next 269 2025-08-28 22:22:49.434546: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fe340000 of size 1310720 next 263 2025-08-28 22:22:49.434553: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fe480000 of size 2359296 next 262 2025-08-28 22:22:49.434560: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fe6c0000 of size 1048576 next 287 2025-08-28 22:22:49.434567: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fe7c0000 of size 1310720 next 281 2025-08-28 22:22:49.434574: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fe900000 of size 2359296 next 280 2025-08-28 22:22:49.434580: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10feb40000 of size 1048576 next 304 2025-08-28 22:22:49.434587: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fec40000 of size 1310720 next 298 2025-08-28 22:22:49.434594: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fed80000 of size 2359296 next 297 2025-08-28 22:22:49.434601: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10fefc0000 of size 1048576 next 322 2025-08-28 22:22:49.434608: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10ff0c0000 of size 1310720 next 316 2025-08-28 22:22:49.434615: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10ff200000 of size 2359296 next 315 2025-08-28 22:22:49.434622: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10ff440000 of size 1048576 next 340 2025-08-28 22:22:49.434629: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10ff540000 of size 1310720 next 334 2025-08-28 22:22:49.434636: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10ff680000 of size 2359296 next 333 2025-08-28 22:22:49.434643: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10ff8c0000 of size 1048576 next 358 2025-08-28 22:22:49.434650: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10ff9c0000 of size 1310720 next 352 2025-08-28 22:22:49.434659: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10ffb00000 of size 2359296 next 351 2025-08-28 22:22:49.434668: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10ffd40000 of size 1048576 next 376 2025-08-28 22:22:49.434680: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f10ffe40000 of size 1835008 next 18446744073709551615 2025-08-28 22:22:49.434695: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 16777216 2025-08-28 22:22:49.434706: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1100000000 of size 2359296 next 181 2025-08-28 22:22:49.434715: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1100240000 of size 2097152 next 196 2025-08-28 22:22:49.434723: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1100440000 of size 1048576 next 216 2025-08-28 22:22:49.434733: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1100540000 of size 1310720 next 209 2025-08-28 22:22:49.434749: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1100680000 of size 2359296 next 208 2025-08-28 22:22:49.434758: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11008c0000 of size 1048576 next 234 2025-08-28 22:22:49.434767: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11009c0000 of size 1310720 next 228 2025-08-28 22:22:49.434776: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1100b00000 of size 2359296 next 227 2025-08-28 22:22:49.434785: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f1100d40000 of size 2883584 next 18446744073709551615 2025-08-28 22:22:49.434794: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 2097152 2025-08-28 22:22:49.434803: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102600000 of size 147456 next 55 2025-08-28 22:22:49.434813: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102624000 of size 65536 next 78 2025-08-28 22:22:49.434822: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102634000 of size 4096 next 191 2025-08-28 22:22:49.434831: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102635000 of size 4096 next 192 2025-08-28 22:22:49.434840: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102636000 of size 4096 next 193 2025-08-28 22:22:49.434850: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102637000 of size 256 next 194 2025-08-28 22:22:49.434859: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102637100 of size 256 next 195 2025-08-28 22:22:49.434868: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102637200 of size 4096 next 197 2025-08-28 22:22:49.434876: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102638200 of size 4096 next 198 2025-08-28 22:22:49.434883: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102639200 of size 4096 next 199 2025-08-28 22:22:49.434892: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110263a200 of size 4096 next 200 2025-08-28 22:22:49.434902: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110263b200 of size 4096 next 201 2025-08-28 22:22:49.434911: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110263c200 of size 1024 next 202 2025-08-28 22:22:49.434920: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110263c600 of size 1024 next 204 2025-08-28 22:22:49.434929: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110263ca00 of size 1024 next 205 2025-08-28 22:22:49.434938: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110263ce00 of size 1024 next 206 2025-08-28 22:22:49.434947: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110263d200 of size 1024 next 207 2025-08-28 22:22:49.434954: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110263d600 of size 1024 next 210 2025-08-28 22:22:49.434961: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110263da00 of size 1024 next 211 2025-08-28 22:22:49.434967: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110263de00 of size 1024 next 212 2025-08-28 22:22:49.434974: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110263e200 of size 1024 next 213 2025-08-28 22:22:49.434981: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110263e600 of size 1024 next 214 2025-08-28 22:22:49.434988: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110263ea00 of size 4096 next 215 2025-08-28 22:22:49.434994: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110263fa00 of size 4096 next 217 2025-08-28 22:22:49.435001: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102640a00 of size 4096 next 218 2025-08-28 22:22:49.435008: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102641a00 of size 4096 next 219 2025-08-28 22:22:49.435021: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102642a00 of size 4096 next 220 2025-08-28 22:22:49.435028: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102643a00 of size 1024 next 221 2025-08-28 22:22:49.435035: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102643e00 of size 1024 next 223 2025-08-28 22:22:49.435042: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102644200 of size 1024 next 224 2025-08-28 22:22:49.435048: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102644600 of size 1024 next 225 2025-08-28 22:22:49.435055: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102644a00 of size 1024 next 226 2025-08-28 22:22:49.435062: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102644e00 of size 1024 next 229 2025-08-28 22:22:49.435069: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102645200 of size 1024 next 230 2025-08-28 22:22:49.435075: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102645600 of size 1024 next 231 2025-08-28 22:22:49.435082: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102645a00 of size 1024 next 232 2025-08-28 22:22:49.435089: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102645e00 of size 1024 next 233 2025-08-28 22:22:49.435096: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102646200 of size 7680 next 72 2025-08-28 22:22:49.435103: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102648000 of size 147456 next 71 2025-08-28 22:22:49.435110: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110266c000 of size 131072 next 87 2025-08-28 22:22:49.435117: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110268c000 of size 4096 next 289 2025-08-28 22:22:49.435124: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110268d000 of size 4096 next 290 2025-08-28 22:22:49.435130: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110268e000 of size 4096 next 291 2025-08-28 22:22:49.435137: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110268f000 of size 1024 next 292 2025-08-28 22:22:49.435144: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110268f400 of size 1024 next 293 2025-08-28 22:22:49.435150: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110268f800 of size 1024 next 294 2025-08-28 22:22:49.435157: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110268fc00 of size 1024 next 295 2025-08-28 22:22:49.435164: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102690000 of size 1024 next 296 2025-08-28 22:22:49.435171: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102690400 of size 1024 next 299 2025-08-28 22:22:49.435177: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102690800 of size 1024 next 300 2025-08-28 22:22:49.435184: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102690c00 of size 1024 next 301 2025-08-28 22:22:49.435191: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102691000 of size 1024 next 302 2025-08-28 22:22:49.435197: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102691400 of size 1024 next 303 2025-08-28 22:22:49.435204: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102691800 of size 4096 next 305 2025-08-28 22:22:49.435211: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102692800 of size 4096 next 306 2025-08-28 22:22:49.435217: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102693800 of size 4096 next 307 2025-08-28 22:22:49.435224: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102694800 of size 4096 next 308 2025-08-28 22:22:49.435231: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102695800 of size 4096 next 309 2025-08-28 22:22:49.435242: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102696800 of size 1024 next 310 2025-08-28 22:22:49.435249: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102696c00 of size 1024 next 311 2025-08-28 22:22:49.435256: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102697000 of size 1024 next 312 2025-08-28 22:22:49.435263: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102697400 of size 1024 next 313 2025-08-28 22:22:49.435269: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102697800 of size 1024 next 314 2025-08-28 22:22:49.435276: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102697c00 of size 1024 next 317 2025-08-28 22:22:49.435283: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102698000 of size 1024 next 318 2025-08-28 22:22:49.435290: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102698400 of size 1024 next 319 2025-08-28 22:22:49.435296: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102698800 of size 1024 next 320 2025-08-28 22:22:49.435303: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102698c00 of size 1024 next 321 2025-08-28 22:22:49.435310: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102699000 of size 4096 next 323 2025-08-28 22:22:49.435317: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110269a000 of size 4096 next 324 2025-08-28 22:22:49.435323: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110269b000 of size 4096 next 325 2025-08-28 22:22:49.435330: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110269c000 of size 4096 next 326 2025-08-28 22:22:49.435337: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110269d000 of size 4096 next 327 2025-08-28 22:22:49.435343: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110269e000 of size 1024 next 328 2025-08-28 22:22:49.435350: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110269e400 of size 1024 next 329 2025-08-28 22:22:49.435357: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110269e800 of size 1024 next 330 2025-08-28 22:22:49.435364: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110269ec00 of size 1024 next 331 2025-08-28 22:22:49.435370: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110269f000 of size 1024 next 332 2025-08-28 22:22:49.435377: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110269f400 of size 1024 next 335 2025-08-28 22:22:49.435384: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110269f800 of size 1024 next 336 2025-08-28 22:22:49.435391: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110269fc00 of size 1024 next 337 2025-08-28 22:22:49.435397: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026a0000 of size 1024 next 338 2025-08-28 22:22:49.435404: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026a0400 of size 1024 next 339 2025-08-28 22:22:49.435411: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026a0800 of size 4096 next 341 2025-08-28 22:22:49.435417: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026a1800 of size 4096 next 342 2025-08-28 22:22:49.435424: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026a2800 of size 4096 next 343 2025-08-28 22:22:49.435431: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026a3800 of size 4096 next 344 2025-08-28 22:22:49.435437: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026a4800 of size 4096 next 345 2025-08-28 22:22:49.435444: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026a5800 of size 1024 next 346 2025-08-28 22:22:49.435451: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026a5c00 of size 1024 next 347 2025-08-28 22:22:49.435458: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026a6000 of size 1024 next 348 2025-08-28 22:22:49.435469: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026a6400 of size 1024 next 349 2025-08-28 22:22:49.435476: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026a6800 of size 1024 next 350 2025-08-28 22:22:49.435483: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026a6c00 of size 1024 next 353 2025-08-28 22:22:49.435489: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026a7000 of size 1024 next 354 2025-08-28 22:22:49.435496: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026a7400 of size 1024 next 355 2025-08-28 22:22:49.435503: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026a7800 of size 1024 next 356 2025-08-28 22:22:49.435510: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026a7c00 of size 1024 next 357 2025-08-28 22:22:49.435516: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026a8000 of size 4096 next 359 2025-08-28 22:22:49.435523: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026a9000 of size 4096 next 360 2025-08-28 22:22:49.435530: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026aa000 of size 4096 next 361 2025-08-28 22:22:49.435537: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026ab000 of size 4096 next 362 2025-08-28 22:22:49.435543: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026ac000 of size 4096 next 363 2025-08-28 22:22:49.435550: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026ad000 of size 1024 next 364 2025-08-28 22:22:49.435557: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026ad400 of size 1024 next 365 2025-08-28 22:22:49.435563: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026ad800 of size 1024 next 366 2025-08-28 22:22:49.435570: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026adc00 of size 1024 next 367 2025-08-28 22:22:49.435577: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026ae000 of size 1024 next 368 2025-08-28 22:22:49.435584: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026ae400 of size 1024 next 371 2025-08-28 22:22:49.435590: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026ae800 of size 1024 next 372 2025-08-28 22:22:49.435597: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026aec00 of size 1024 next 373 2025-08-28 22:22:49.435604: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026af000 of size 1024 next 374 2025-08-28 22:22:49.435611: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026af400 of size 1024 next 375 2025-08-28 22:22:49.435617: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026af800 of size 4096 next 377 2025-08-28 22:22:49.435624: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026b0800 of size 4096 next 378 2025-08-28 22:22:49.435631: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026b1800 of size 4096 next 379 2025-08-28 22:22:49.435637: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026b2800 of size 4096 next 380 2025-08-28 22:22:49.435644: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026b3800 of size 4096 next 381 2025-08-28 22:22:49.435651: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026b4800 of size 1024 next 382 2025-08-28 22:22:49.435658: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026b4c00 of size 1024 next 383 2025-08-28 22:22:49.435664: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026b5000 of size 1024 next 384 2025-08-28 22:22:49.435671: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026b5400 of size 1024 next 385 2025-08-28 22:22:49.435678: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026b5800 of size 1024 next 386 2025-08-28 22:22:49.435689: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026b5c00 of size 1024 next 389 2025-08-28 22:22:49.435696: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026b6000 of size 1024 next 390 2025-08-28 22:22:49.435703: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026b6400 of size 1024 next 391 2025-08-28 22:22:49.435709: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026b6800 of size 1024 next 392 2025-08-28 22:22:49.435716: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026b6c00 of size 1024 next 393 2025-08-28 22:22:49.435723: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026b7000 of size 4096 next 395 2025-08-28 22:22:49.435730: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026b8000 of size 4096 next 396 2025-08-28 22:22:49.435736: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026b9000 of size 4096 next 397 2025-08-28 22:22:49.435743: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026ba000 of size 4096 next 398 2025-08-28 22:22:49.435750: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026bb000 of size 4096 next 399 2025-08-28 22:22:49.435756: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026bc000 of size 1024 next 400 2025-08-28 22:22:49.435763: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026bc400 of size 1024 next 401 2025-08-28 22:22:49.435770: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026bc800 of size 1024 next 402 2025-08-28 22:22:49.435777: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026bcc00 of size 1024 next 403 2025-08-28 22:22:49.435783: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026bd000 of size 1024 next 404 2025-08-28 22:22:49.435790: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026bd400 of size 1024 next 407 2025-08-28 22:22:49.435797: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026bd800 of size 1024 next 408 2025-08-28 22:22:49.435803: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026bdc00 of size 1024 next 409 2025-08-28 22:22:49.435810: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026be000 of size 1024 next 410 2025-08-28 22:22:49.435817: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026be400 of size 1024 next 411 2025-08-28 22:22:49.435824: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026be800 of size 4096 next 413 2025-08-28 22:22:49.435830: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026bf800 of size 4096 next 414 2025-08-28 22:22:49.435837: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026c0800 of size 4096 next 415 2025-08-28 22:22:49.435844: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026c1800 of size 4096 next 416 2025-08-28 22:22:49.435851: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026c2800 of size 4096 next 417 2025-08-28 22:22:49.435857: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026c3800 of size 1024 next 418 2025-08-28 22:22:49.435864: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026c3c00 of size 1024 next 419 2025-08-28 22:22:49.435871: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026c4000 of size 1024 next 420 2025-08-28 22:22:49.435878: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026c4400 of size 1024 next 421 2025-08-28 22:22:49.435884: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026c4800 of size 1024 next 422 2025-08-28 22:22:49.435891: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026c4c00 of size 1024 next 425 2025-08-28 22:22:49.435898: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026c5000 of size 1024 next 426 2025-08-28 22:22:49.435909: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026c5400 of size 1024 next 427 2025-08-28 22:22:49.435916: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026c5800 of size 1024 next 428 2025-08-28 22:22:49.435923: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026c5c00 of size 1024 next 429 2025-08-28 22:22:49.435930: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026c6000 of size 4096 next 431 2025-08-28 22:22:49.435936: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026c7000 of size 4096 next 432 2025-08-28 22:22:49.435943: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026c8000 of size 4096 next 433 2025-08-28 22:22:49.435950: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026c9000 of size 4096 next 434 2025-08-28 22:22:49.435956: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026ca000 of size 4096 next 435 2025-08-28 22:22:49.435963: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026cb000 of size 1024 next 436 2025-08-28 22:22:49.435970: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026cb400 of size 1024 next 437 2025-08-28 22:22:49.435977: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026cb800 of size 1024 next 438 2025-08-28 22:22:49.435983: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026cbc00 of size 1024 next 103 2025-08-28 22:22:49.435990: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11026cc000 of size 262144 next 102 2025-08-28 22:22:49.435997: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f110270c000 of size 999424 next 18446744073709551615 2025-08-28 22:22:49.436004: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 4194304 2025-08-28 22:22:49.436011: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102800000 of size 589824 next 96 2025-08-28 22:22:49.436018: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102890000 of size 262144 next 117 2025-08-28 22:22:49.436025: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028d0000 of size 1024 next 439 2025-08-28 22:22:49.436032: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028d0400 of size 1024 next 442 2025-08-28 22:22:49.436038: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028d0800 of size 1024 next 443 2025-08-28 22:22:49.436045: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028d0c00 of size 1024 next 444 2025-08-28 22:22:49.436052: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028d1000 of size 1024 next 445 2025-08-28 22:22:49.436059: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028d1400 of size 1024 next 446 2025-08-28 22:22:49.436065: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028d1800 of size 4096 next 448 2025-08-28 22:22:49.436072: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028d2800 of size 4096 next 449 2025-08-28 22:22:49.436079: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028d3800 of size 4096 next 450 2025-08-28 22:22:49.436086: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028d4800 of size 4096 next 451 2025-08-28 22:22:49.436092: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028d5800 of size 4096 next 452 2025-08-28 22:22:49.436099: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028d6800 of size 1024 next 453 2025-08-28 22:22:49.436106: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028d6c00 of size 1024 next 454 2025-08-28 22:22:49.436113: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028d7000 of size 1024 next 455 2025-08-28 22:22:49.436119: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028d7400 of size 1024 next 456 2025-08-28 22:22:49.436131: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028d7800 of size 1024 next 457 2025-08-28 22:22:49.436138: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028d7c00 of size 1024 next 460 2025-08-28 22:22:49.436145: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028d8000 of size 1024 next 461 2025-08-28 22:22:49.436151: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028d8400 of size 1024 next 462 2025-08-28 22:22:49.436158: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028d8800 of size 1024 next 463 2025-08-28 22:22:49.436165: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028d8c00 of size 1024 next 464 2025-08-28 22:22:49.436172: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028d9000 of size 4096 next 466 2025-08-28 22:22:49.436178: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028da000 of size 4096 next 467 2025-08-28 22:22:49.436185: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028db000 of size 4096 next 468 2025-08-28 22:22:49.436192: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028dc000 of size 4096 next 469 2025-08-28 22:22:49.436198: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028dd000 of size 4096 next 470 2025-08-28 22:22:49.436205: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028de000 of size 1024 next 471 2025-08-28 22:22:49.436212: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028de400 of size 1024 next 472 2025-08-28 22:22:49.436219: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028de800 of size 1024 next 473 2025-08-28 22:22:49.436225: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028dec00 of size 1024 next 474 2025-08-28 22:22:49.436232: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028df000 of size 1024 next 475 2025-08-28 22:22:49.436239: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028df400 of size 1024 next 478 2025-08-28 22:22:49.436246: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028df800 of size 1024 next 479 2025-08-28 22:22:49.436252: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028dfc00 of size 1024 next 480 2025-08-28 22:22:49.436259: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028e0000 of size 1024 next 481 2025-08-28 22:22:49.436266: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028e0400 of size 1024 next 482 2025-08-28 22:22:49.436272: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028e0800 of size 4096 next 484 2025-08-28 22:22:49.436279: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028e1800 of size 4096 next 485 2025-08-28 22:22:49.436286: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028e2800 of size 4096 next 486 2025-08-28 22:22:49.436293: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028e3800 of size 4096 next 487 2025-08-28 22:22:49.436312: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028e4800 of size 4096 next 488 2025-08-28 22:22:49.436319: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028e5800 of size 1024 next 489 2025-08-28 22:22:49.436326: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028e5c00 of size 1024 next 490 2025-08-28 22:22:49.436333: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028e6000 of size 1024 next 491 2025-08-28 22:22:49.436359: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028e6400 of size 1024 next 492 2025-08-28 22:22:49.436369: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028e6800 of size 1024 next 493 2025-08-28 22:22:49.436376: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028e6c00 of size 1024 next 496 2025-08-28 22:22:49.436382: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028e7000 of size 1024 next 497 2025-08-28 22:22:49.436397: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028e7400 of size 1024 next 498 2025-08-28 22:22:49.436404: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028e7800 of size 1024 next 499 2025-08-28 22:22:49.436411: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028e7c00 of size 1024 next 500 2025-08-28 22:22:49.436418: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028e8000 of size 4096 next 502 2025-08-28 22:22:49.436425: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028e9000 of size 4096 next 503 2025-08-28 22:22:49.436431: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028ea000 of size 4096 next 504 2025-08-28 22:22:49.436438: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028eb000 of size 4096 next 505 2025-08-28 22:22:49.436445: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028ec000 of size 4096 next 506 2025-08-28 22:22:49.436452: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028ed000 of size 1024 next 507 2025-08-28 22:22:49.436458: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028ed400 of size 1024 next 508 2025-08-28 22:22:49.436465: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028ed800 of size 1024 next 509 2025-08-28 22:22:49.436472: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028edc00 of size 1024 next 510 2025-08-28 22:22:49.436479: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028ee000 of size 1024 next 511 2025-08-28 22:22:49.436486: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028ee400 of size 1024 next 514 2025-08-28 22:22:49.436492: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028ee800 of size 1024 next 515 2025-08-28 22:22:49.436499: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028eec00 of size 1024 next 516 2025-08-28 22:22:49.436506: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028ef000 of size 1024 next 517 2025-08-28 22:22:49.436513: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028ef400 of size 1024 next 518 2025-08-28 22:22:49.436519: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028ef800 of size 4096 next 520 2025-08-28 22:22:49.436526: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028f0800 of size 4096 next 521 2025-08-28 22:22:49.436533: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028f1800 of size 4096 next 522 2025-08-28 22:22:49.436540: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028f2800 of size 4096 next 523 2025-08-28 22:22:49.436546: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028f3800 of size 4096 next 524 2025-08-28 22:22:49.436553: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028f4800 of size 1024 next 525 2025-08-28 22:22:49.436560: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028f4c00 of size 1024 next 526 2025-08-28 22:22:49.436567: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028f5000 of size 1024 next 527 2025-08-28 22:22:49.436573: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028f5400 of size 1024 next 528 2025-08-28 22:22:49.436580: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028f5800 of size 1024 next 529 2025-08-28 22:22:49.436587: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028f5c00 of size 1024 next 532 2025-08-28 22:22:49.436594: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028f6000 of size 1024 next 533 2025-08-28 22:22:49.436601: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028f6400 of size 1024 next 534 2025-08-28 22:22:49.436607: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028f6800 of size 1024 next 535 2025-08-28 22:22:49.436618: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028f6c00 of size 1024 next 536 2025-08-28 22:22:49.436625: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028f7000 of size 4096 next 538 2025-08-28 22:22:49.436632: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028f8000 of size 4096 next 539 2025-08-28 22:22:49.436639: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028f9000 of size 4096 next 540 2025-08-28 22:22:49.436646: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028fa000 of size 4096 next 541 2025-08-28 22:22:49.436653: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028fb000 of size 4096 next 542 2025-08-28 22:22:49.436659: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028fc000 of size 1024 next 543 2025-08-28 22:22:49.436666: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028fc400 of size 1024 next 544 2025-08-28 22:22:49.436673: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028fc800 of size 1024 next 545 2025-08-28 22:22:49.436680: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028fcc00 of size 1024 next 546 2025-08-28 22:22:49.436686: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028fd000 of size 1024 next 547 2025-08-28 22:22:49.436693: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028fd400 of size 1024 next 550 2025-08-28 22:22:49.436700: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028fd800 of size 1024 next 551 2025-08-28 22:22:49.436707: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028fdc00 of size 1024 next 552 2025-08-28 22:22:49.436713: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028fe000 of size 1024 next 553 2025-08-28 22:22:49.436720: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028fe400 of size 1024 next 554 2025-08-28 22:22:49.436727: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028fe800 of size 4096 next 556 2025-08-28 22:22:49.436734: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11028ff800 of size 4096 next 557 2025-08-28 22:22:49.436740: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102900800 of size 4096 next 558 2025-08-28 22:22:49.436747: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102901800 of size 4096 next 559 2025-08-28 22:22:49.436754: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102902800 of size 4096 next 560 2025-08-28 22:22:49.436761: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102903800 of size 1024 next 561 2025-08-28 22:22:49.436767: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102903c00 of size 1024 next 562 2025-08-28 22:22:49.436774: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102904000 of size 1024 next 563 2025-08-28 22:22:49.436781: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102904400 of size 1024 next 564 2025-08-28 22:22:49.436788: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102904800 of size 1024 next 565 2025-08-28 22:22:49.436794: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102904c00 of size 1024 next 568 2025-08-28 22:22:49.436801: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102905000 of size 1024 next 569 2025-08-28 22:22:49.436808: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102905400 of size 1024 next 570 2025-08-28 22:22:49.436815: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102905800 of size 1024 next 571 2025-08-28 22:22:49.436821: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102905c00 of size 1024 next 572 2025-08-28 22:22:49.436828: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102906000 of size 4096 next 574 2025-08-28 22:22:49.436839: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102907000 of size 4096 next 575 2025-08-28 22:22:49.436846: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102908000 of size 4096 next 576 2025-08-28 22:22:49.436853: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102909000 of size 4096 next 577 2025-08-28 22:22:49.436860: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110290a000 of size 4096 next 578 2025-08-28 22:22:49.436867: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110290b000 of size 1024 next 579 2025-08-28 22:22:49.436873: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110290b400 of size 1024 next 580 2025-08-28 22:22:49.436880: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110290b800 of size 1024 next 581 2025-08-28 22:22:49.436887: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110290bc00 of size 1024 next 582 2025-08-28 22:22:49.436894: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110290c000 of size 1024 next 583 2025-08-28 22:22:49.436901: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110290c400 of size 1024 next 586 2025-08-28 22:22:49.436908: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110290c800 of size 1024 next 587 2025-08-28 22:22:49.436914: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110290cc00 of size 1024 next 588 2025-08-28 22:22:49.436921: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110290d000 of size 1024 next 589 2025-08-28 22:22:49.436928: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110290d400 of size 1024 next 590 2025-08-28 22:22:49.436935: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110290d800 of size 4096 next 592 2025-08-28 22:22:49.436942: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f110290e800 of size 6144 next 112 2025-08-28 22:22:49.436949: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102910000 of size 524288 next 111 2025-08-28 22:22:49.436956: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102990000 of size 262144 next 135 2025-08-28 22:22:49.436963: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11029d0000 of size 327680 next 124 2025-08-28 22:22:49.436970: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102a20000 of size 589824 next 123 2025-08-28 22:22:49.436977: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102ab0000 of size 262144 next 152 2025-08-28 22:22:49.436984: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102af0000 of size 327680 next 142 2025-08-28 22:22:49.436991: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102b40000 of size 786432 next 18446744073709551615 2025-08-28 22:22:49.436998: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 8388608 2025-08-28 22:22:49.437005: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c00000 of size 4096 next 593 2025-08-28 22:22:49.437012: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c01000 of size 4096 next 594 2025-08-28 22:22:49.437019: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c02000 of size 4096 next 595 2025-08-28 22:22:49.437026: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c03000 of size 2048 next 598 2025-08-28 22:22:49.437033: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c03800 of size 2048 next 599 2025-08-28 22:22:49.437039: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c04000 of size 2048 next 600 2025-08-28 22:22:49.437046: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c04800 of size 2048 next 601 2025-08-28 22:22:49.437053: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c05000 of size 2048 next 602 2025-08-28 22:22:49.437064: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c05800 of size 256 next 605 2025-08-28 22:22:49.437071: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c05900 of size 256 next 606 2025-08-28 22:22:49.437078: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c05a00 of size 2048 next 604 2025-08-28 22:22:49.437085: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c06200 of size 2048 next 609 2025-08-28 22:22:49.437092: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c06a00 of size 2048 next 610 2025-08-28 22:22:49.437099: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c07200 of size 2048 next 611 2025-08-28 22:22:49.437105: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c07a00 of size 2048 next 612 2025-08-28 22:22:49.437112: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c08200 of size 256 next 614 2025-08-28 22:22:49.437119: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c08300 of size 256 next 615 2025-08-28 22:22:49.437129: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c08400 of size 8192 next 613 2025-08-28 22:22:49.437143: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c0a400 of size 8192 next 618 2025-08-28 22:22:49.437155: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c0c400 of size 8192 next 619 2025-08-28 22:22:49.437167: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c0e400 of size 8192 next 620 2025-08-28 22:22:49.437179: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c10400 of size 8192 next 621 2025-08-28 22:22:49.437191: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c12400 of size 256 next 622 2025-08-28 22:22:49.437203: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c12500 of size 256 next 623 2025-08-28 22:22:49.437215: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c12600 of size 8192 next 626 2025-08-28 22:22:49.437226: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c14600 of size 8192 next 627 2025-08-28 22:22:49.437238: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c16600 of size 8192 next 628 2025-08-28 22:22:49.437250: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c18600 of size 8192 next 629 2025-08-28 22:22:49.437262: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c1a600 of size 8192 next 630 2025-08-28 22:22:49.437274: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c1c600 of size 2048 next 631 2025-08-28 22:22:49.437286: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c1ce00 of size 2048 next 633 2025-08-28 22:22:49.437298: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c1d600 of size 2048 next 634 2025-08-28 22:22:49.437309: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c1de00 of size 2048 next 635 2025-08-28 22:22:49.437321: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c1e600 of size 2048 next 636 2025-08-28 22:22:49.437333: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f1102c1ee00 of size 463360 next 160 2025-08-28 22:22:49.437345: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102c90000 of size 589824 next 158 2025-08-28 22:22:49.437357: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102d20000 of size 524288 next 171 2025-08-28 22:22:49.437369: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f1102da0000 of size 1048576 next 222 2025-08-28 22:22:49.437381: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102ea0000 of size 1048576 next 190 2025-08-28 22:22:49.437393: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f1102fa0000 of size 1048576 next 189 2025-08-28 22:22:49.437417: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11030a0000 of size 1048576 next 203 2025-08-28 22:22:49.437430: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11031a0000 of size 1048576 next 251 2025-08-28 22:22:49.437442: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f11032a0000 of size 1441792 next 18446744073709551615 2025-08-28 22:22:49.437454: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 1048576 2025-08-28 22:22:49.437466: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce00000 of size 1280 next 1 2025-08-28 22:22:49.437478: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce00500 of size 256 next 5 2025-08-28 22:22:49.437490: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce00600 of size 256 next 8 2025-08-28 22:22:49.437502: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce00700 of size 256 next 9 2025-08-28 22:22:49.437514: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce00800 of size 256 next 10 2025-08-28 22:22:49.437526: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce00900 of size 256 next 11 2025-08-28 22:22:49.437538: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce00a00 of size 256 next 12 2025-08-28 22:22:49.437550: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce00b00 of size 256 next 13 2025-08-28 22:22:49.437561: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce00c00 of size 256 next 17 2025-08-28 22:22:49.437573: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce00d00 of size 256 next 19 2025-08-28 22:22:49.437585: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce00e00 of size 256 next 20 2025-08-28 22:22:49.437597: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce00f00 of size 256 next 21 2025-08-28 22:22:49.437609: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce01000 of size 256 next 22 2025-08-28 22:22:49.437620: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce01100 of size 256 next 24 2025-08-28 22:22:49.437632: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce01200 of size 256 next 25 2025-08-28 22:22:49.437644: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce01300 of size 256 next 23 2025-08-28 22:22:49.437656: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce01400 of size 256 next 28 2025-08-28 22:22:49.437668: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce01500 of size 256 next 29 2025-08-28 22:22:49.437679: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce01600 of size 256 next 30 2025-08-28 22:22:49.437691: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce01700 of size 256 next 31 2025-08-28 22:22:49.437703: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce01800 of size 256 next 33 2025-08-28 22:22:49.437715: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce01900 of size 256 next 34 2025-08-28 22:22:49.437727: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce01a00 of size 1024 next 32 2025-08-28 22:22:49.437739: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce01e00 of size 1024 next 37 2025-08-28 22:22:49.437750: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce02200 of size 1024 next 38 2025-08-28 22:22:49.437762: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce02600 of size 1024 next 39 2025-08-28 22:22:49.437774: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce02a00 of size 1024 next 40 2025-08-28 22:22:49.437786: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce02e00 of size 1024 next 41 2025-08-28 22:22:49.437798: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce03200 of size 1024 next 43 2025-08-28 22:22:49.437818: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce03600 of size 1024 next 44 2025-08-28 22:22:49.437830: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce03a00 of size 1024 next 45 2025-08-28 22:22:49.437842: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce03e00 of size 1024 next 46 2025-08-28 22:22:49.437854: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce04200 of size 256 next 48 2025-08-28 22:22:49.437865: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce04300 of size 256 next 49 2025-08-28 22:22:49.437877: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce04400 of size 256 next 50 2025-08-28 22:22:49.437889: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce04500 of size 256 next 51 2025-08-28 22:22:49.437901: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce04600 of size 256 next 52 2025-08-28 22:22:49.437913: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce04700 of size 256 next 53 2025-08-28 22:22:49.437925: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce04800 of size 256 next 56 2025-08-28 22:22:49.437936: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce04900 of size 256 next 57 2025-08-28 22:22:49.437948: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce04a00 of size 256 next 58 2025-08-28 22:22:49.437960: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce04b00 of size 256 next 14 2025-08-28 22:22:49.437972: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce04c00 of size 256 next 15 2025-08-28 22:22:49.437984: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce04d00 of size 256 next 16 2025-08-28 22:22:49.437996: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce04e00 of size 1024 next 60 2025-08-28 22:22:49.438007: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce05200 of size 1024 next 61 2025-08-28 22:22:49.438019: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce05600 of size 1024 next 62 2025-08-28 22:22:49.438031: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce05a00 of size 1024 next 63 2025-08-28 22:22:49.438043: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce05e00 of size 1024 next 64 2025-08-28 22:22:49.438055: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce06200 of size 256 next 66 2025-08-28 22:22:49.438066: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce06300 of size 256 next 67 2025-08-28 22:22:49.438078: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce06400 of size 256 next 68 2025-08-28 22:22:49.438090: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce06500 of size 256 next 69 2025-08-28 22:22:49.438102: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce06600 of size 256 next 70 2025-08-28 22:22:49.438114: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce06700 of size 256 next 73 2025-08-28 22:22:49.438126: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce06800 of size 256 next 74 2025-08-28 22:22:49.438138: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce06900 of size 256 next 75 2025-08-28 22:22:49.438149: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce06a00 of size 256 next 76 2025-08-28 22:22:49.438161: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce06b00 of size 256 next 77 2025-08-28 22:22:49.438173: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce06c00 of size 1024 next 79 2025-08-28 22:22:49.438185: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce07000 of size 1024 next 80 2025-08-28 22:22:49.438197: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce07400 of size 1024 next 81 2025-08-28 22:22:49.438216: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce07800 of size 1024 next 82 2025-08-28 22:22:49.438228: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce07c00 of size 1024 next 83 2025-08-28 22:22:49.438240: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce08000 of size 256 next 85 2025-08-28 22:22:49.438252: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce08100 of size 256 next 86 2025-08-28 22:22:49.438264: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce08200 of size 512 next 84 2025-08-28 22:22:49.438276: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce08400 of size 512 next 88 2025-08-28 22:22:49.438288: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce08600 of size 512 next 89 2025-08-28 22:22:49.438300: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce08800 of size 512 next 90 2025-08-28 22:22:49.438312: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce08a00 of size 512 next 91 2025-08-28 22:22:49.438324: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce08c00 of size 256 next 93 2025-08-28 22:22:49.438335: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce08d00 of size 256 next 94 2025-08-28 22:22:49.438347: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce08e00 of size 512 next 92 2025-08-28 22:22:49.438359: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce09000 of size 512 next 97 2025-08-28 22:22:49.438371: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce09200 of size 512 next 98 2025-08-28 22:22:49.438383: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce09400 of size 512 next 99 2025-08-28 22:22:49.438395: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce09600 of size 512 next 2 2025-08-28 22:22:49.438406: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce09800 of size 256 next 3 2025-08-28 22:22:49.438418: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce09900 of size 256 next 4 2025-08-28 22:22:49.438431: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce09a00 of size 16384 next 18 2025-08-28 22:22:49.438443: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce0da00 of size 256 next 100 2025-08-28 22:22:49.438454: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce0db00 of size 256 next 101 2025-08-28 22:22:49.438466: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce0dc00 of size 2048 next 104 2025-08-28 22:22:49.438478: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce0e400 of size 2048 next 105 2025-08-28 22:22:49.438490: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce0ec00 of size 2048 next 106 2025-08-28 22:22:49.438502: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce0f400 of size 2048 next 107 2025-08-28 22:22:49.438514: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce0fc00 of size 2048 next 108 2025-08-28 22:22:49.438525: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce10400 of size 256 next 109 2025-08-28 22:22:49.438537: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce10500 of size 256 next 110 2025-08-28 22:22:49.438549: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce10600 of size 2048 next 113 2025-08-28 22:22:49.438561: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce10e00 of size 2048 next 114 2025-08-28 22:22:49.438573: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce11600 of size 2048 next 115 2025-08-28 22:22:49.438585: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce11e00 of size 2048 next 116 2025-08-28 22:22:49.438605: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce12600 of size 2048 next 6 2025-08-28 22:22:49.438617: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce12e00 of size 37632 next 7 2025-08-28 22:22:49.438629: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce1c100 of size 512 next 118 2025-08-28 22:22:49.438641: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce1c300 of size 512 next 119 2025-08-28 22:22:49.438653: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce1c500 of size 512 next 120 2025-08-28 22:22:49.438665: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce1c700 of size 512 next 121 2025-08-28 22:22:49.438677: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce1c900 of size 512 next 122 2025-08-28 22:22:49.438688: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce1cb00 of size 512 next 125 2025-08-28 22:22:49.438700: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce1cd00 of size 512 next 126 2025-08-28 22:22:49.438712: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce1cf00 of size 512 next 127 2025-08-28 22:22:49.438724: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce1d100 of size 512 next 128 2025-08-28 22:22:49.438736: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce1d300 of size 512 next 129 2025-08-28 22:22:49.438747: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce1d500 of size 2048 next 130 2025-08-28 22:22:49.438759: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce1dd00 of size 2048 next 131 2025-08-28 22:22:49.438771: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce1e500 of size 2048 next 132 2025-08-28 22:22:49.438783: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce1ed00 of size 2048 next 133 2025-08-28 22:22:49.438795: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce1f500 of size 2048 next 134 2025-08-28 22:22:49.438807: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce1fd00 of size 512 next 136 2025-08-28 22:22:49.438819: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce1ff00 of size 512 next 137 2025-08-28 22:22:49.438830: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce20100 of size 512 next 138 2025-08-28 22:22:49.438842: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce20300 of size 512 next 139 2025-08-28 22:22:49.438854: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce20500 of size 512 next 140 2025-08-28 22:22:49.438866: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce20700 of size 512 next 141 2025-08-28 22:22:49.438878: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce20900 of size 512 next 143 2025-08-28 22:22:49.438889: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce20b00 of size 512 next 144 2025-08-28 22:22:49.438901: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce20d00 of size 512 next 145 2025-08-28 22:22:49.438913: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce20f00 of size 512 next 146 2025-08-28 22:22:49.438925: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce21100 of size 2048 next 147 2025-08-28 22:22:49.438937: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce21900 of size 2048 next 148 2025-08-28 22:22:49.438948: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce22100 of size 2048 next 149 2025-08-28 22:22:49.438960: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce22900 of size 2048 next 150 2025-08-28 22:22:49.438972: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce23100 of size 2048 next 151 2025-08-28 22:22:49.438984: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce23900 of size 512 next 153 2025-08-28 22:22:49.439004: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce23b00 of size 512 next 154 2025-08-28 22:22:49.439016: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce23d00 of size 512 next 155 2025-08-28 22:22:49.439028: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce23f00 of size 512 next 156 2025-08-28 22:22:49.439040: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce24100 of size 512 next 157 2025-08-28 22:22:49.439051: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce24300 of size 512 next 161 2025-08-28 22:22:49.439063: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce24500 of size 512 next 162 2025-08-28 22:22:49.439075: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce24700 of size 512 next 163 2025-08-28 22:22:49.439087: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce24900 of size 512 next 164 2025-08-28 22:22:49.439098: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce24b00 of size 512 next 165 2025-08-28 22:22:49.439110: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce24d00 of size 2048 next 166 2025-08-28 22:22:49.439122: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce25500 of size 2048 next 167 2025-08-28 22:22:49.439134: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce25d00 of size 2048 next 168 2025-08-28 22:22:49.439146: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce26500 of size 2048 next 169 2025-08-28 22:22:49.439158: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce26d00 of size 2048 next 170 2025-08-28 22:22:49.439169: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce27500 of size 1024 next 172 2025-08-28 22:22:49.439181: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce27900 of size 1024 next 173 2025-08-28 22:22:49.439193: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce27d00 of size 1024 next 174 2025-08-28 22:22:49.439205: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce28100 of size 1024 next 175 2025-08-28 22:22:49.439217: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce28500 of size 1024 next 176 2025-08-28 22:22:49.439229: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce28900 of size 256 next 178 2025-08-28 22:22:49.439241: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce28a00 of size 256 next 179 2025-08-28 22:22:49.439252: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce28b00 of size 1024 next 177 2025-08-28 22:22:49.439264: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce28f00 of size 1024 next 182 2025-08-28 22:22:49.439276: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce29300 of size 1024 next 183 2025-08-28 22:22:49.439288: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce29700 of size 1024 next 184 2025-08-28 22:22:49.439300: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce29b00 of size 1024 next 185 2025-08-28 22:22:49.439311: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce29f00 of size 256 next 187 2025-08-28 22:22:49.439323: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce2a000 of size 256 next 188 2025-08-28 22:22:49.439335: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce2a100 of size 4096 next 186 2025-08-28 22:22:49.439347: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce2b100 of size 4096 next 42 2025-08-28 22:22:49.439359: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce2c100 of size 65536 next 36 2025-08-28 22:22:49.439371: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce3c100 of size 65536 next 35 2025-08-28 22:22:49.439390: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce4c100 of size 4096 next 235 2025-08-28 22:22:49.439403: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce4d100 of size 4096 next 236 2025-08-28 22:22:49.439414: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce4e100 of size 4096 next 237 2025-08-28 22:22:49.439443: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce4f100 of size 4096 next 238 2025-08-28 22:22:49.439456: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce50100 of size 1024 next 239 2025-08-28 22:22:49.439468: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce50500 of size 1024 next 240 2025-08-28 22:22:49.439480: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce50900 of size 1024 next 241 2025-08-28 22:22:49.439492: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce50d00 of size 1024 next 242 2025-08-28 22:22:49.439504: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce51100 of size 1024 next 243 2025-08-28 22:22:49.439516: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce51500 of size 1024 next 246 2025-08-28 22:22:49.439527: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce51900 of size 1024 next 247 2025-08-28 22:22:49.439539: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce51d00 of size 1024 next 248 2025-08-28 22:22:49.439551: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce52100 of size 1024 next 249 2025-08-28 22:22:49.439563: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce52500 of size 1024 next 250 2025-08-28 22:22:49.439575: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce52900 of size 4096 next 252 2025-08-28 22:22:49.439587: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce53900 of size 4096 next 253 2025-08-28 22:22:49.439599: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce54900 of size 4096 next 254 2025-08-28 22:22:49.439611: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce55900 of size 4096 next 255 2025-08-28 22:22:49.439622: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce56900 of size 4096 next 256 2025-08-28 22:22:49.439634: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce57900 of size 1024 next 257 2025-08-28 22:22:49.439646: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce57d00 of size 1024 next 258 2025-08-28 22:22:49.439658: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce58100 of size 1024 next 259 2025-08-28 22:22:49.439670: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce58500 of size 1024 next 260 2025-08-28 22:22:49.439682: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce58900 of size 1024 next 261 2025-08-28 22:22:49.439694: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce58d00 of size 1024 next 264 2025-08-28 22:22:49.439705: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce59100 of size 1024 next 265 2025-08-28 22:22:49.439717: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce59500 of size 1024 next 266 2025-08-28 22:22:49.439729: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce59900 of size 1024 next 267 2025-08-28 22:22:49.439741: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce59d00 of size 1024 next 268 2025-08-28 22:22:49.439753: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce5a100 of size 4096 next 270 2025-08-28 22:22:49.439765: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce5b100 of size 4096 next 271 2025-08-28 22:22:49.439818: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce5c100 of size 4096 next 272 2025-08-28 22:22:49.439851: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce5d100 of size 4096 next 273 2025-08-28 22:22:49.439867: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce5e100 of size 4096 next 274 2025-08-28 22:22:49.439883: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce5f100 of size 1024 next 275 2025-08-28 22:22:49.439899: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce5f500 of size 1024 next 276 2025-08-28 22:22:49.439914: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce5f900 of size 1024 next 277 2025-08-28 22:22:49.439930: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce5fd00 of size 1024 next 278 2025-08-28 22:22:49.439946: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce60100 of size 1024 next 279 2025-08-28 22:22:49.439961: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce60500 of size 1024 next 282 2025-08-28 22:22:49.439977: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce60900 of size 1024 next 283 2025-08-28 22:22:49.439993: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce60d00 of size 1024 next 284 2025-08-28 22:22:49.440009: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce61100 of size 1024 next 285 2025-08-28 22:22:49.440025: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce61500 of size 1024 next 286 2025-08-28 22:22:49.440038: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce61900 of size 4096 next 288 2025-08-28 22:22:49.440051: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce62900 of size 6144 next 27 2025-08-28 22:22:49.440062: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce64100 of size 147456 next 26 2025-08-28 22:22:49.440074: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce88100 of size 65536 next 47 2025-08-28 22:22:49.440086: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ce98100 of size 65536 next 59 2025-08-28 22:22:49.440098: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123cea8100 of size 65536 next 65 2025-08-28 22:22:49.440111: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f123ceb8100 of size 294656 next 18446744073709551615 2025-08-28 22:22:49.440122: I tensorflow/core/common_runtime/bfc_allocator.cc:995] Summary of in-use Chunks by size: 2025-08-28 22:22:49.440145: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 65 Chunks of size 256 totalling 16.2KiB 2025-08-28 22:22:49.440159: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 40 Chunks of size 512 totalling 20.0KiB 2025-08-28 22:22:49.440172: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 250 Chunks of size 1024 totalling 250.0KiB 2025-08-28 22:22:49.440185: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1280 totalling 1.2KiB 2025-08-28 22:22:49.440199: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 40 Chunks of size 2048 totalling 80.0KiB 2025-08-28 22:22:49.440212: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 117 Chunks of size 4096 totalling 468.0KiB 2025-08-28 22:22:49.440225: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 6144 totalling 12.0KiB 2025-08-28 22:22:49.440238: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 7680 totalling 7.5KiB 2025-08-28 22:22:49.440251: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 10 Chunks of size 8192 totalling 80.0KiB 2025-08-28 22:22:49.440264: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 16384 totalling 16.0KiB 2025-08-28 22:22:49.440278: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 37632 totalling 36.8KiB 2025-08-28 22:22:49.440291: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 6 Chunks of size 65536 totalling 384.0KiB 2025-08-28 22:22:49.440304: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 131072 totalling 128.0KiB 2025-08-28 22:22:49.440328: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 147456 totalling 432.0KiB 2025-08-28 22:22:49.440357: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 4 Chunks of size 262144 totalling 1.00MiB 2025-08-28 22:22:49.440374: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 294656 totalling 287.8KiB 2025-08-28 22:22:49.440387: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 327680 totalling 640.0KiB 2025-08-28 22:22:49.440400: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 524288 totalling 1.00MiB 2025-08-28 22:22:49.440413: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 589824 totalling 1.69MiB 2025-08-28 22:22:49.440426: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 786432 totalling 768.0KiB 2025-08-28 22:22:49.440440: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 25 Chunks of size 1048576 totalling 25.00MiB 2025-08-28 22:22:49.440453: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 18 Chunks of size 1310720 totalling 22.50MiB 2025-08-28 22:22:49.440466: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1441792 totalling 1.38MiB 2025-08-28 22:22:49.440479: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1835008 totalling 1.75MiB 2025-08-28 22:22:49.440492: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 2097152 totalling 4.00MiB 2025-08-28 22:22:49.440505: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 23 Chunks of size 2359296 totalling 51.75MiB 2025-08-28 22:22:49.440518: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 4194304 totalling 8.00MiB 2025-08-28 22:22:49.440531: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 8388608 totalling 8.00MiB 2025-08-28 22:22:49.440544: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 9437184 totalling 18.00MiB 2025-08-28 22:22:49.440558: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 11993088 totalling 11.44MiB 2025-08-28 22:22:49.440571: I tensorflow/core/common_runtime/bfc_allocator.cc:1002] Sum Total of in-use chunks: 159.04MiB 2025-08-28 22:22:49.440583: I tensorflow/core/common_runtime/bfc_allocator.cc:1004] total_region_allocated_bytes_: 199688192 memory_limit_: 199688192 available bytes: 0 curr_region_allocation_bytes_: 268435456 2025-08-28 22:22:49.440600: I tensorflow/core/common_runtime/bfc_allocator.cc:1010] Stats: Limit: 199688192 InUse: 166768128 MaxInUse: 166768384 NumAllocs: 2113 MaxAllocSize: 11993088 2025-08-28 22:22:49.440635: W tensorflow/core/common_runtime/bfc_allocator.cc:439] __*****__******___***********************************************_*************************_******** 2025-08-28 22:22:49.440684: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at cwise_ops_common.h:134 : Resource exhausted: OOM when allocating tensor with shape[3,3,512,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl3453 thcls : [{'id': 3453, 'mtr_user_id': 31, 'name': 'mask_refus_amiens_050123', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4485, 'photo_desc_type': 5732, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}] thcl {'id': 3453, 'mtr_user_id': 31, 'name': 'mask_refus_amiens_050123', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4485, 'photo_desc_type': 5732, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5732 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5732, 'mask_refus_amiens_050123', 16384, 25088, 'mask_refus_amiens_050123', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 1, 5, 13, 23, 59), datetime.datetime(2023, 1, 5, 13, 23, 59)) {'thcl': {'id': 3453, 'mtr_user_id': 31, 'name': 'mask_refus_amiens_050123', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4485, 'photo_desc_type': 5732, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'], 'list_hashtags_csv': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4485, 'svm_hashtag_type_desc': 5732, 'photo_desc_type': 5732, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_refus_amiens_050123 NUM_CLASSES 9 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 Exception in mask_detect : OOM when allocating tensor with shape[3,3,512,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Add] we want to redo the detection Using TensorFlow backend. max_time_sub_proc : 3600 erreur pendant la detection Catched exception ! (2013, 'Lost connection to MySQL server during query') Connect or reconnect ! in case -12 caffe_path_current : About to save ! 2 After save, about to update current ! 24.52user 27.94system 1:24.30elapsed 62%CPU (0avgtext+0avgdata 2563412maxresident)k 1912392inputs+16144outputs (13633major+2266493minor)pagefaults 0swaps