python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' -a 4741 -l 30 ' -s datou_current_4741 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2981259 load datou : 4741 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 13156 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 13160 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 13160 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 13157 crop_condition is not consistent : 4 used against 2 in the step definition ! WARNING : number of outputs for step 13157 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 13161 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 13159 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 13159 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 13156 doesn't seem to be define in the database( WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 13161 doesn't seem to be define in the database( WARNING : type of input 3 of step 13159 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13163 have datatype=6 whereas input 2 of step 13157 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13162 have datatype=6 whereas input 2 of step 13157 have datatype=None We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 13161 have datatype=10 whereas input 3 of step 13164 have datatype=6 WARNING : output 0 of step 13161 have datatype=10 whereas input 0 of step 13166 have datatype=18 WARNING : type of input 5 of step 13164 doesn't seem to be define in the database( WARNING : output 0 of step 13166 have datatype=11 whereas input 5 of step 13164 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 1 of step 13158 have datatype=7 whereas input 2 of step 13157 have datatype=None WARNING : type of output 3 of step 13157 doesn't seem to be define in the database( WARNING : type of input 3 of step 13161 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13163 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13162 doesn't seem to be define in the database( DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5732, 'mask_refus_amiens_050123', 16384, 25088, 'mask_refus_amiens_050123', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 1, 5, 13, 23, 59), datetime.datetime(2023, 1, 5, 13, 23, 59)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5972, 'learn_entrant_syctomXV_111023', 16384, 25088, 'learn_entrant_syctomXV_111023', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 10, 11, 15, 57, 35), datetime.datetime(2023, 10, 11, 15, 57, 35)) load thcls load THCL from format json or kwargs add thcl : 3453 in CacheModelConfig load THCL from format json or kwargs add thcl : 3783 in CacheModelConfig load pdts add pdt : 5732 in CacheModelConfig add pdt : 5972 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec no input labels no input values updating current state to 1 list_input_json: {} Current got : datou_id : 4741, datou_cur_ids : ['3763069'] with mtr_portfolio_ids : ['27124915'] and first list_photo_ids : [] new path : /proc/2981259/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 13156 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 13160 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 13160 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 13157 crop_condition is not consistent : 4 used against 2 in the step definition ! WARNING : number of outputs for step 13157 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 13161 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 13159 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 13159 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 13156 doesn't seem to be define in the database( WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 13161 doesn't seem to be define in the database( WARNING : type of input 3 of step 13159 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13163 have datatype=6 whereas input 2 of step 13157 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13162 have datatype=6 whereas input 2 of step 13157 have datatype=None We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 13161 have datatype=10 whereas input 3 of step 13164 have datatype=6 WARNING : output 0 of step 13161 have datatype=10 whereas input 0 of step 13166 have datatype=18 WARNING : type of input 5 of step 13164 doesn't seem to be define in the database( WARNING : output 0 of step 13166 have datatype=11 whereas input 5 of step 13164 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 1 of step 13158 have datatype=7 whereas input 2 of step 13157 have datatype=None WARNING : type of output 3 of step 13157 doesn't seem to be define in the database( WARNING : type of input 3 of step 13161 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13163 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13162 doesn't seem to be define in the database( DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, mask_detect, blur_detection, brightness, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : {} origin We have 1 , BFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 8 ; length of list_pids : 8 ; length of list_args : 8 time to download the photos : 2.2853877544403076 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Tue Sep 23 11:21:31 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 9585 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-09-23 11:21:33.931784: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-09-23 11:21:33.956506: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3492910000 Hz 2025-09-23 11:21:33.958874: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f8890000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-09-23 11:21:33.958919: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-09-23 11:21:33.962330: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-09-23 11:21:34.121934: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x12eb82d0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-09-23 11:21:34.122005: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-09-23 11:21:34.123418: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-09-23 11:21:34.123948: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-23 11:21:34.130431: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-23 11:21:34.134250: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-09-23 11:21:34.134626: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-09-23 11:21:34.137612: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-09-23 11:21:34.139006: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-09-23 11:21:34.144414: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-23 11:21:34.145635: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-09-23 11:21:34.145707: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-23 11:21:34.146338: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-09-23 11:21:34.146354: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-09-23 11:21:34.146363: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-09-23 11:21:34.147482: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4833 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-09-23 11:21:34.430391: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-09-23 11:21:34.430488: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-23 11:21:34.430510: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-23 11:21:34.430529: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-09-23 11:21:34.430547: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-09-23 11:21:34.430565: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-09-23 11:21:34.430583: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-09-23 11:21:34.430602: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-23 11:21:34.431690: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-09-23 11:21:34.432801: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-09-23 11:21:34.432840: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-23 11:21:34.432860: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-23 11:21:34.432879: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-09-23 11:21:34.432908: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-09-23 11:21:34.432927: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-09-23 11:21:34.432945: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-09-23 11:21:34.432964: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-23 11:21:34.434100: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-09-23 11:21:34.434130: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-09-23 11:21:34.434141: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-09-23 11:21:34.434150: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-09-23 11:21:34.435375: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4833 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl3783 thcls : [{'id': 3783, 'mtr_user_id': 31, 'name': 'learn_entrant_syctomXV_111023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'photo_desc_type': 5972, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 3783, 'mtr_user_id': 31, 'name': 'learn_entrant_syctomXV_111023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'photo_desc_type': 5972, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5972 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5972, 'learn_entrant_syctomXV_111023', 16384, 25088, 'learn_entrant_syctomXV_111023', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 10, 11, 15, 57, 35), datetime.datetime(2023, 10, 11, 15, 57, 35)) {'thcl': {'id': 3783, 'mtr_user_id': 31, 'name': 'learn_entrant_syctomXV_111023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'photo_desc_type': 5972, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'sac', 'textiles', 'verre', 'organique', 'dasri', 'masque', 'encombrant', 'autre_emballage', 'autre_non_emballage', 'environnement'], 'list_hashtags_csv': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'svm_hashtag_type_desc': 5972, 'photo_desc_type': 5972, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'sac', 'textiles', 'verre', 'organique', 'dasri', 'masque', 'encombrant', 'autre_emballage', 'autre_non_emballage', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_entrant_syctomXV_111023 NUM_CLASSES 11 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_entrant_syctomXV_111023 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-09-23 11:21:45.826548: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-23 11:21:45.997112: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_entrant_syctomXV_111023 /data/models_weight/learn_entrant_syctomXV_111023/mask_model.h5 size_local : 256052544 size in s3 : 256052544 create time local : 2023-10-26 16:36:43 create time in s3 : 2023-10-11 13:57:17 mask_model.h5 already exist and didn't need to update list_images length : 8 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.68047 max: 151.10000 image_metas shape: (1, 19) min: 0.00000 max: 3120.00000 nb d'objets trouves : 26 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 19) min: 0.00000 max: 3120.00000 nb d'objets trouves : 15 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 19) min: 0.00000 max: 3120.00000 nb d'objets trouves : 22 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.08047 image_metas shape: (1, 19) min: 0.00000 max: 3120.00000 nb d'objets trouves : 13 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 19) min: 0.00000 max: 3120.00000 nb d'objets trouves : 19 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 19) min: 0.00000 max: 3120.00000 nb d'objets trouves : 17 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 19) min: 0.00000 max: 3120.00000 nb d'objets trouves : 19 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 19) min: 0.00000 max: 3120.00000 nb d'objets trouves : 6 Detection mask done ! Trying to reset tf kernel 2981556 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 1196 tf kernel not reseted sub process len(results) : 8 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 8 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 6485 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl3783 Catched exception ! Connect or reconnect ! thcls : [{'id': 3783, 'mtr_user_id': 31, 'name': 'learn_entrant_syctomXV_111023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'photo_desc_type': 5972, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 3783, 'mtr_user_id': 31, 'name': 'learn_entrant_syctomXV_111023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'photo_desc_type': 5972, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5972 ['background', 'sac', 'textiles', 'verre', 'organique', 'dasri', 'masque', 'encombrant', 'autre_emballage', 'autre_non_emballage', 'environnement'] time for calcul the mask position with numpy : 0.01956963539123535 nb_pixel_total : 447297 time to create 1 rle with new method : 0.027792930603027344 length of segment : 710 time for calcul the mask position with numpy : 0.004969120025634766 nb_pixel_total : 326220 time to create 1 rle with new method : 0.017467498779296875 length of segment : 825 time for calcul the mask position with numpy : 0.0008993148803710938 nb_pixel_total : 55963 time to create 1 rle with old method : 0.06485486030578613 length of segment : 302 time for calcul the mask position with numpy : 0.0019292831420898438 nb_pixel_total : 108870 time to create 1 rle with old method : 0.14668536186218262 length of segment : 326 time for calcul the mask position with numpy : 0.0004215240478515625 nb_pixel_total : 24761 time to create 1 rle with old method : 0.027865886688232422 length of segment : 187 time for calcul the mask position with numpy : 0.0010654926300048828 nb_pixel_total : 52187 time to create 1 rle with old method : 0.07435297966003418 length of segment : 380 time for calcul the mask position with numpy : 0.0003790855407714844 nb_pixel_total : 17049 time to create 1 rle with old method : 0.018947601318359375 length of segment : 294 time for calcul the mask position with numpy : 0.0008025169372558594 nb_pixel_total : 45550 time to create 1 rle with old method : 0.050731658935546875 length of segment : 352 time for calcul the mask position with numpy : 0.0004093647003173828 nb_pixel_total : 22032 time to create 1 rle with old method : 0.030651092529296875 length of segment : 256 time for calcul the mask position with numpy : 0.0010051727294921875 nb_pixel_total : 54827 time to create 1 rle with old method : 0.08005928993225098 length of segment : 264 time for calcul the mask position with numpy : 0.00029349327087402344 nb_pixel_total : 14826 time to create 1 rle with old method : 0.017009973526000977 length of segment : 116 time for calcul the mask position with numpy : 0.26867198944091797 nb_pixel_total : 1027702 time to create 1 rle with new method : 0.09889864921569824 length of segment : 1159 time for calcul the mask position with numpy : 0.0004916191101074219 nb_pixel_total : 18117 time to create 1 rle with old method : 0.019597768783569336 length of segment : 373 time for calcul the mask position with numpy : 0.00023484230041503906 nb_pixel_total : 7650 time to create 1 rle with old method : 0.009917497634887695 length of segment : 88 time for calcul the mask position with numpy : 0.0012750625610351562 nb_pixel_total : 77914 time to create 1 rle with old method : 0.09043383598327637 length of segment : 352 time for calcul the mask position with numpy : 0.00043129920959472656 nb_pixel_total : 17763 time to create 1 rle with old method : 0.019467830657958984 length of segment : 234 time for calcul the mask position with numpy : 0.002711057662963867 nb_pixel_total : 151733 time to create 1 rle with new method : 0.007893562316894531 length of segment : 794 time for calcul the mask position with numpy : 0.0011074542999267578 nb_pixel_total : 50036 time to create 1 rle with old method : 0.055579185485839844 length of segment : 393 time for calcul the mask position with numpy : 0.0038330554962158203 nb_pixel_total : 124708 time to create 1 rle with old method : 0.24528717994689941 length of segment : 466 time for calcul the mask position with numpy : 0.0015997886657714844 nb_pixel_total : 42021 time to create 1 rle with old method : 0.07820534706115723 length of segment : 298 time for calcul the mask position with numpy : 0.0004963874816894531 nb_pixel_total : 17225 time to create 1 rle with old method : 0.03138470649719238 length of segment : 121 time for calcul the mask position with numpy : 0.0022857189178466797 nb_pixel_total : 82900 time to create 1 rle with old method : 0.13149690628051758 length of segment : 355 time for calcul the mask position with numpy : 0.0011012554168701172 nb_pixel_total : 60254 time to create 1 rle with old method : 0.06829404830932617 length of segment : 364 time for calcul the mask position with numpy : 0.0005321502685546875 nb_pixel_total : 19404 time to create 1 rle with old method : 0.029230833053588867 length of segment : 195 time for calcul the mask position with numpy : 0.00030112266540527344 nb_pixel_total : 13807 time to create 1 rle with old method : 0.016545772552490234 length of segment : 190 time for calcul the mask position with numpy : 0.0010058879852294922 nb_pixel_total : 25495 time to create 1 rle with old method : 0.031071901321411133 length of segment : 173 time for calcul the mask position with numpy : 0.00853109359741211 nb_pixel_total : 334353 time to create 1 rle with new method : 0.012196779251098633 length of segment : 674 time for calcul the mask position with numpy : 0.0013682842254638672 nb_pixel_total : 45528 time to create 1 rle with old method : 0.06011009216308594 length of segment : 248 time for calcul the mask position with numpy : 0.0005354881286621094 nb_pixel_total : 20015 time to create 1 rle with old method : 0.026699066162109375 length of segment : 84 time for calcul the mask position with numpy : 0.0018334388732910156 nb_pixel_total : 41656 time to create 1 rle with old method : 0.05434441566467285 length of segment : 345 time for calcul the mask position with numpy : 0.03576183319091797 nb_pixel_total : 1301589 time to create 1 rle with new method : 1.750197172164917 length of segment : 1374 time for calcul the mask position with numpy : 0.0002651214599609375 nb_pixel_total : 10027 time to create 1 rle with old method : 0.011581897735595703 length of segment : 168 time for calcul the mask position with numpy : 0.0011937618255615234 nb_pixel_total : 50923 time to create 1 rle with old method : 0.059372663497924805 length of segment : 253 time for calcul the mask position with numpy : 0.0008378028869628906 nb_pixel_total : 27281 time to create 1 rle with old method : 0.03155159950256348 length of segment : 193 time for calcul the mask position with numpy : 0.0018014907836914062 nb_pixel_total : 55583 time to create 1 rle with old method : 0.06254720687866211 length of segment : 562 time for calcul the mask position with numpy : 0.0005879402160644531 nb_pixel_total : 19165 time to create 1 rle with old method : 0.022607088088989258 length of segment : 192 time for calcul the mask position with numpy : 0.0015075206756591797 nb_pixel_total : 77831 time to create 1 rle with old method : 0.08652091026306152 length of segment : 605 time for calcul the mask position with numpy : 0.00023627281188964844 nb_pixel_total : 5051 time to create 1 rle with old method : 0.0057523250579833984 length of segment : 85 time for calcul the mask position with numpy : 0.0005869865417480469 nb_pixel_total : 20620 time to create 1 rle with old method : 0.02389693260192871 length of segment : 112 time for calcul the mask position with numpy : 0.0004878044128417969 nb_pixel_total : 8490 time to create 1 rle with old method : 0.009618520736694336 length of segment : 216 time for calcul the mask position with numpy : 0.0007407665252685547 nb_pixel_total : 17009 time to create 1 rle with old method : 0.019781827926635742 length of segment : 184 time for calcul the mask position with numpy : 0.0019457340240478516 nb_pixel_total : 43483 time to create 1 rle with old method : 0.048612117767333984 length of segment : 367 time for calcul the mask position with numpy : 0.004042863845825195 nb_pixel_total : 124091 time to create 1 rle with old method : 0.14109230041503906 length of segment : 381 time for calcul the mask position with numpy : 0.011305570602416992 nb_pixel_total : 273524 time to create 1 rle with new method : 0.021341800689697266 length of segment : 805 time for calcul the mask position with numpy : 0.0005776882171630859 nb_pixel_total : 16894 time to create 1 rle with old method : 0.023283720016479492 length of segment : 116 time for calcul the mask position with numpy : 0.0022003650665283203 nb_pixel_total : 63903 time to create 1 rle with old method : 0.08434343338012695 length of segment : 254 time for calcul the mask position with numpy : 0.0006401538848876953 nb_pixel_total : 10528 time to create 1 rle with old method : 0.012196540832519531 length of segment : 134 time for calcul the mask position with numpy : 0.0029497146606445312 nb_pixel_total : 76840 time to create 1 rle with old method : 0.08648252487182617 length of segment : 412 time for calcul the mask position with numpy : 0.0002586841583251953 nb_pixel_total : 11243 time to create 1 rle with old method : 0.012902498245239258 length of segment : 152 time for calcul the mask position with numpy : 0.001043081283569336 nb_pixel_total : 28766 time to create 1 rle with old method : 0.0325465202331543 length of segment : 198 time for calcul the mask position with numpy : 0.004735469818115234 nb_pixel_total : 100034 time to create 1 rle with old method : 0.1126396656036377 length of segment : 507 time for calcul the mask position with numpy : 0.023650169372558594 nb_pixel_total : 441226 time to create 1 rle with new method : 0.320831298828125 length of segment : 604 time for calcul the mask position with numpy : 0.0006673336029052734 nb_pixel_total : 9245 time to create 1 rle with old method : 0.01076960563659668 length of segment : 160 time for calcul the mask position with numpy : 0.0002841949462890625 nb_pixel_total : 6571 time to create 1 rle with old method : 0.007737636566162109 length of segment : 86 time for calcul the mask position with numpy : 0.0005140304565429688 nb_pixel_total : 20209 time to create 1 rle with old method : 0.023426532745361328 length of segment : 157 time for calcul the mask position with numpy : 0.0004506111145019531 nb_pixel_total : 5628 time to create 1 rle with old method : 0.006846427917480469 length of segment : 170 time for calcul the mask position with numpy : 0.0010242462158203125 nb_pixel_total : 35821 time to create 1 rle with old method : 0.04217028617858887 length of segment : 246 time for calcul the mask position with numpy : 0.00043463706970214844 nb_pixel_total : 10145 time to create 1 rle with old method : 0.012079715728759766 length of segment : 168 time for calcul the mask position with numpy : 0.0028247833251953125 nb_pixel_total : 46062 time to create 1 rle with old method : 0.05392599105834961 length of segment : 318 time for calcul the mask position with numpy : 0.0010900497436523438 nb_pixel_total : 6740 time to create 1 rle with old method : 0.008199453353881836 length of segment : 108 time for calcul the mask position with numpy : 0.0018548965454101562 nb_pixel_total : 47101 time to create 1 rle with old method : 0.05529022216796875 length of segment : 415 time for calcul the mask position with numpy : 0.0027856826782226562 nb_pixel_total : 62496 time to create 1 rle with old method : 0.07246112823486328 length of segment : 277 time for calcul the mask position with numpy : 0.0004591941833496094 nb_pixel_total : 21387 time to create 1 rle with old method : 0.024334192276000977 length of segment : 182 time for calcul the mask position with numpy : 0.0038733482360839844 nb_pixel_total : 137251 time to create 1 rle with old method : 0.15366530418395996 length of segment : 548 time for calcul the mask position with numpy : 0.0033202171325683594 nb_pixel_total : 160667 time to create 1 rle with new method : 0.004599332809448242 length of segment : 368 time for calcul the mask position with numpy : 0.0015306472778320312 nb_pixel_total : 59302 time to create 1 rle with old method : 0.09071826934814453 length of segment : 456 time for calcul the mask position with numpy : 0.0030825138092041016 nb_pixel_total : 160205 time to create 1 rle with new method : 0.004895925521850586 length of segment : 415 time for calcul the mask position with numpy : 0.005852699279785156 nb_pixel_total : 267058 time to create 1 rle with new method : 0.008955001831054688 length of segment : 334 time for calcul the mask position with numpy : 0.002875089645385742 nb_pixel_total : 122278 time to create 1 rle with old method : 0.13876628875732422 length of segment : 567 time for calcul the mask position with numpy : 0.0007655620574951172 nb_pixel_total : 33321 time to create 1 rle with old method : 0.03840970993041992 length of segment : 171 time for calcul the mask position with numpy : 0.0015521049499511719 nb_pixel_total : 46051 time to create 1 rle with old method : 0.05283641815185547 length of segment : 431 time for calcul the mask position with numpy : 0.0005557537078857422 nb_pixel_total : 18095 time to create 1 rle with old method : 0.020884990692138672 length of segment : 149 time for calcul the mask position with numpy : 0.0005819797515869141 nb_pixel_total : 22810 time to create 1 rle with old method : 0.026722192764282227 length of segment : 125 time for calcul the mask position with numpy : 0.0005486011505126953 nb_pixel_total : 3298 time to create 1 rle with old method : 0.0041925907135009766 length of segment : 133 time for calcul the mask position with numpy : 0.0005087852478027344 nb_pixel_total : 18484 time to create 1 rle with old method : 0.021005868911743164 length of segment : 177 time for calcul the mask position with numpy : 0.0002868175506591797 nb_pixel_total : 6899 time to create 1 rle with old method : 0.007795810699462891 length of segment : 91 time for calcul the mask position with numpy : 0.0004012584686279297 nb_pixel_total : 12940 time to create 1 rle with old method : 0.015257835388183594 length of segment : 140 time for calcul the mask position with numpy : 0.001615762710571289 nb_pixel_total : 62781 time to create 1 rle with old method : 0.0697934627532959 length of segment : 226 time for calcul the mask position with numpy : 0.0009503364562988281 nb_pixel_total : 27230 time to create 1 rle with old method : 0.030957698822021484 length of segment : 172 time for calcul the mask position with numpy : 0.0011584758758544922 nb_pixel_total : 50079 time to create 1 rle with old method : 0.05810999870300293 length of segment : 238 time for calcul the mask position with numpy : 0.0006506443023681641 nb_pixel_total : 25296 time to create 1 rle with old method : 0.029900550842285156 length of segment : 124 time for calcul the mask position with numpy : 0.0008268356323242188 nb_pixel_total : 5230 time to create 1 rle with old method : 0.006404399871826172 length of segment : 166 time for calcul the mask position with numpy : 0.0018160343170166016 nb_pixel_total : 87561 time to create 1 rle with old method : 0.09845709800720215 length of segment : 360 time for calcul the mask position with numpy : 0.0003254413604736328 nb_pixel_total : 7700 time to create 1 rle with old method : 0.009671688079833984 length of segment : 97 time for calcul the mask position with numpy : 0.0013387203216552734 nb_pixel_total : 45978 time to create 1 rle with old method : 0.051674842834472656 length of segment : 249 time for calcul the mask position with numpy : 0.0008270740509033203 nb_pixel_total : 27259 time to create 1 rle with old method : 0.03162384033203125 length of segment : 151 time for calcul the mask position with numpy : 0.0010094642639160156 nb_pixel_total : 43644 time to create 1 rle with old method : 0.05049562454223633 length of segment : 155 time for calcul the mask position with numpy : 0.000354766845703125 nb_pixel_total : 24889 time to create 1 rle with old method : 0.02828526496887207 length of segment : 118 time for calcul the mask position with numpy : 0.0008189678192138672 nb_pixel_total : 21803 time to create 1 rle with old method : 0.025190114974975586 length of segment : 174 time for calcul the mask position with numpy : 0.0002713203430175781 nb_pixel_total : 6111 time to create 1 rle with old method : 0.00701594352722168 length of segment : 87 time for calcul the mask position with numpy : 0.0007281303405761719 nb_pixel_total : 12974 time to create 1 rle with old method : 0.015778779983520508 length of segment : 190 time for calcul the mask position with numpy : 0.0005595684051513672 nb_pixel_total : 41830 time to create 1 rle with old method : 0.04836750030517578 length of segment : 149 time for calcul the mask position with numpy : 0.00031495094299316406 nb_pixel_total : 9373 time to create 1 rle with old method : 0.015603780746459961 length of segment : 106 time for calcul the mask position with numpy : 0.0010814666748046875 nb_pixel_total : 44318 time to create 1 rle with old method : 0.0680849552154541 length of segment : 272 time for calcul the mask position with numpy : 0.0006601810455322266 nb_pixel_total : 44292 time to create 1 rle with old method : 0.05098390579223633 length of segment : 162 time for calcul the mask position with numpy : 0.0042667388916015625 nb_pixel_total : 162446 time to create 1 rle with new method : 0.0067255496978759766 length of segment : 449 time for calcul the mask position with numpy : 0.0030126571655273438 nb_pixel_total : 95143 time to create 1 rle with old method : 0.10645532608032227 length of segment : 636 time for calcul the mask position with numpy : 0.0004818439483642578 nb_pixel_total : 11154 time to create 1 rle with old method : 0.012772560119628906 length of segment : 142 time for calcul the mask position with numpy : 0.000850677490234375 nb_pixel_total : 21137 time to create 1 rle with old method : 0.024434566497802734 length of segment : 359 time for calcul the mask position with numpy : 0.0008573532104492188 nb_pixel_total : 27882 time to create 1 rle with old method : 0.033257246017456055 length of segment : 186 time for calcul the mask position with numpy : 0.0015325546264648438 nb_pixel_total : 39570 time to create 1 rle with old method : 0.04434633255004883 length of segment : 322 time for calcul the mask position with numpy : 0.0002524852752685547 nb_pixel_total : 8677 time to create 1 rle with old method : 0.009898900985717773 length of segment : 144 time for calcul the mask position with numpy : 0.0005414485931396484 nb_pixel_total : 1541 time to create 1 rle with old method : 0.002105236053466797 length of segment : 52 time for calcul the mask position with numpy : 0.0002579689025878906 nb_pixel_total : 9662 time to create 1 rle with old method : 0.011208534240722656 length of segment : 78 time for calcul the mask position with numpy : 0.0007510185241699219 nb_pixel_total : 38448 time to create 1 rle with old method : 0.04362773895263672 length of segment : 305 time for calcul the mask position with numpy : 0.00032019615173339844 nb_pixel_total : 4819 time to create 1 rle with old method : 0.005533456802368164 length of segment : 138 time for calcul the mask position with numpy : 0.0014138221740722656 nb_pixel_total : 54897 time to create 1 rle with old method : 0.06278061866760254 length of segment : 332 time for calcul the mask position with numpy : 0.0005252361297607422 nb_pixel_total : 15293 time to create 1 rle with old method : 0.017915725708007812 length of segment : 155 time for calcul the mask position with numpy : 0.0009253025054931641 nb_pixel_total : 31960 time to create 1 rle with old method : 0.03660774230957031 length of segment : 269 time for calcul the mask position with numpy : 0.0012347698211669922 nb_pixel_total : 27072 time to create 1 rle with old method : 0.03172039985656738 length of segment : 344 time for calcul the mask position with numpy : 0.0007350444793701172 nb_pixel_total : 21357 time to create 1 rle with old method : 0.02439594268798828 length of segment : 242 time for calcul the mask position with numpy : 0.00029087066650390625 nb_pixel_total : 15539 time to create 1 rle with old method : 0.017960548400878906 length of segment : 167 time for calcul the mask position with numpy : 0.0009267330169677734 nb_pixel_total : 25636 time to create 1 rle with old method : 0.02997446060180664 length of segment : 309 time for calcul the mask position with numpy : 0.0018558502197265625 nb_pixel_total : 65015 time to create 1 rle with old method : 0.07503867149353027 length of segment : 403 time for calcul the mask position with numpy : 0.008702278137207031 nb_pixel_total : 247084 time to create 1 rle with new method : 0.014559030532836914 length of segment : 835 time for calcul the mask position with numpy : 0.0006883144378662109 nb_pixel_total : 15203 time to create 1 rle with old method : 0.017589807510375977 length of segment : 220 time for calcul the mask position with numpy : 0.02097773551940918 nb_pixel_total : 532446 time to create 1 rle with new method : 0.08962297439575195 length of segment : 1137 time for calcul the mask position with numpy : 0.004883766174316406 nb_pixel_total : 180482 time to create 1 rle with new method : 0.007477998733520508 length of segment : 696 time for calcul the mask position with numpy : 0.0010676383972167969 nb_pixel_total : 27490 time to create 1 rle with old method : 0.0431363582611084 length of segment : 274 time for calcul the mask position with numpy : 0.0004420280456542969 nb_pixel_total : 3524 time to create 1 rle with old method : 0.004300355911254883 length of segment : 117 time for calcul the mask position with numpy : 0.011047840118408203 nb_pixel_total : 545389 time to create 1 rle with new method : 0.04972195625305176 length of segment : 1356 time for calcul the mask position with numpy : 0.0008397102355957031 nb_pixel_total : 28568 time to create 1 rle with old method : 0.03233170509338379 length of segment : 213 time for calcul the mask position with numpy : 0.002361774444580078 nb_pixel_total : 96030 time to create 1 rle with old method : 0.11218023300170898 length of segment : 324 time for calcul the mask position with numpy : 0.0007998943328857422 nb_pixel_total : 12711 time to create 1 rle with old method : 0.015122652053833008 length of segment : 271 time for calcul the mask position with numpy : 0.0001595020294189453 nb_pixel_total : 3915 time to create 1 rle with old method : 0.004786014556884766 length of segment : 89 time for calcul the mask position with numpy : 0.00029754638671875 nb_pixel_total : 11293 time to create 1 rle with old method : 0.013602256774902344 length of segment : 166 time for calcul the mask position with numpy : 0.0006549358367919922 nb_pixel_total : 31188 time to create 1 rle with old method : 0.03609466552734375 length of segment : 268 time for calcul the mask position with numpy : 0.000370025634765625 nb_pixel_total : 8010 time to create 1 rle with old method : 0.009230613708496094 length of segment : 137 time for calcul the mask position with numpy : 0.00024008750915527344 nb_pixel_total : 5815 time to create 1 rle with old method : 0.0071332454681396484 length of segment : 159 time for calcul the mask position with numpy : 0.0005600452423095703 nb_pixel_total : 6659 time to create 1 rle with old method : 0.00867772102355957 length of segment : 141 time for calcul the mask position with numpy : 0.0007119178771972656 nb_pixel_total : 18069 time to create 1 rle with old method : 0.020770788192749023 length of segment : 265 time for calcul the mask position with numpy : 0.016908884048461914 nb_pixel_total : 568885 time to create 1 rle with new method : 0.3555731773376465 length of segment : 2595 time for calcul the mask position with numpy : 0.6328227519989014 nb_pixel_total : 3366634 time to create 1 rle with new method : 0.7229127883911133 length of segment : 1911 time for calcul the mask position with numpy : 0.003744840621948242 nb_pixel_total : 205646 time to create 1 rle with new method : 0.007361650466918945 length of segment : 738 time for calcul the mask position with numpy : 0.09124207496643066 nb_pixel_total : 2342486 time to create 1 rle with new method : 0.40917086601257324 length of segment : 1516 time for calcul the mask position with numpy : 0.39407896995544434 nb_pixel_total : 3937718 time to create 1 rle with new method : 0.6626989841461182 length of segment : 1825 time for calcul the mask position with numpy : 0.029096126556396484 nb_pixel_total : 2580355 time to create 1 rle with new method : 0.11924481391906738 length of segment : 1246 time spent for convertir_results : 22.377256393432617 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 137 chid ids of type : 4854 Number RLEs to save : 0 save missing photos in datou_result : time spend for datou_step_exec : 73.0043957233429 time spend to save output : 0.13094401359558105 total time spend for step 1 : 73.13533973693848 step2:mask_detect Tue Sep 23 11:22:44 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 2784 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-09-23 11:22:46.906077: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-09-23 11:22:46.932557: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3492910000 Hz 2025-09-23 11:22:46.934497: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f8878000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-09-23 11:22:46.934545: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-09-23 11:22:46.937969: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-09-23 11:22:47.065385: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x12df27f0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-09-23 11:22:47.065430: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-09-23 11:22:47.066314: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-09-23 11:22:47.066709: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-23 11:22:47.102991: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-23 11:22:47.105641: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-09-23 11:22:47.106102: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-09-23 11:22:47.109062: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-09-23 11:22:47.110641: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-09-23 11:22:47.115217: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-23 11:22:47.116127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-09-23 11:22:47.116187: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-23 11:22:47.116732: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-09-23 11:22:47.116748: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-09-23 11:22:47.116757: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-09-23 11:22:47.117656: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2256 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-09-23 11:22:47.282170: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-09-23 11:22:47.282272: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-23 11:22:47.282296: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-23 11:22:47.282317: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-09-23 11:22:47.282337: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-09-23 11:22:47.282357: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-09-23 11:22:47.282377: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-09-23 11:22:47.282397: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-23 11:22:47.283373: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-09-23 11:22:47.284355: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-09-23 11:22:47.284393: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-09-23 11:22:47.284415: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-09-23 11:22:47.284434: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-09-23 11:22:47.284453: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-09-23 11:22:47.284472: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-09-23 11:22:47.284492: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-09-23 11:22:47.284511: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-09-23 11:22:47.285479: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-09-23 11:22:47.285509: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-09-23 11:22:47.285520: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-09-23 11:22:47.285529: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-09-23 11:22:47.286541: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2256 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) 2025-09-23 11:22:47.984882: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 16.00M (16777216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-23 11:22:47.985447: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 14.40M (15099648 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-23 11:22:47.985959: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 12.96M (13589760 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-23 11:22:47.986464: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 11.66M (12230912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-23 11:22:47.986959: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 10.50M (11008000 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-23 11:22:47.987441: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 9.45M (9907200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-23 11:22:47.987910: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 8.50M (8916480 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-23 11:22:47.988400: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 7.65M (8024832 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-23 11:22:47.988929: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 6.89M (7222528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-23 11:22:47.989458: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 6.20M (6500352 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-23 11:22:47.990009: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 5.58M (5850368 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-23 11:22:47.990518: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 5.02M (5265408 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-23 11:22:47.991029: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.52M (4739072 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-23 11:22:47.991540: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.07M (4265216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-23 11:22:48.017355: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-23 11:22:48.017982: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-23 11:22:58.018990: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-23 11:22:58.019836: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 32.00M (33554432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-09-23 11:22:58.019867: W tensorflow/core/common_runtime/bfc_allocator.cc:434] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.00MiB (rounded to 2097152) Current allocation summary follows. 2025-09-23 11:22:58.019886: I tensorflow/core/common_runtime/bfc_allocator.cc:934] BFCAllocator dump for GPU_0_bfc 2025-09-23 11:22:58.019904: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (256): Total Chunks: 59, Chunks in use: 59. 14.8KiB allocated for chunks. 14.8KiB in use in bin. 8.8KiB client-requested in use in bin. 2025-09-23 11:22:58.019920: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (512): Total Chunks: 40, Chunks in use: 40. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin. 2025-09-23 11:22:58.019935: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1024): Total Chunks: 31, Chunks in use: 31. 31.2KiB allocated for chunks. 31.2KiB in use in bin. 31.0KiB client-requested in use in bin. 2025-09-23 11:22:58.019950: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2048): Total Chunks: 25, Chunks in use: 25. 50.0KiB allocated for chunks. 50.0KiB in use in bin. 50.0KiB client-requested in use in bin. 2025-09-23 11:22:58.019965: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4096): Total Chunks: 5, Chunks in use: 5. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin. 2025-09-23 11:22:58.019979: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-09-23 11:22:58.019994: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16384): Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin. 2025-09-23 11:22:58.020025: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (32768): Total Chunks: 1, Chunks in use: 1. 36.8KiB allocated for chunks. 36.8KiB in use in bin. 36.8KiB client-requested in use in bin. 2025-09-23 11:22:58.020041: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (65536): Total Chunks: 8, Chunks in use: 6. 547.5KiB allocated for chunks. 384.0KiB in use in bin. 384.0KiB client-requested in use in bin. 2025-09-23 11:22:58.020056: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (131072): Total Chunks: 4, Chunks in use: 4. 560.0KiB allocated for chunks. 560.0KiB in use in bin. 560.0KiB client-requested in use in bin. 2025-09-23 11:22:58.020071: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (262144): Total Chunks: 9, Chunks in use: 7. 2.41MiB allocated for chunks. 1.91MiB in use in bin. 1.75MiB client-requested in use in bin. 2025-09-23 11:22:58.020086: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (524288): Total Chunks: 8, Chunks in use: 6. 4.95MiB allocated for chunks. 3.44MiB in use in bin. 3.25MiB client-requested in use in bin. 2025-09-23 11:22:58.020100: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1048576): Total Chunks: 1, Chunks in use: 1. 1.00MiB allocated for chunks. 1.00MiB in use in bin. 1.00MiB client-requested in use in bin. 2025-09-23 11:22:58.020115: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2097152): Total Chunks: 3, Chunks in use: 3. 9.04MiB allocated for chunks. 9.04MiB in use in bin. 6.25MiB client-requested in use in bin. 2025-09-23 11:22:58.020128: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-09-23 11:22:58.020141: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-09-23 11:22:58.020154: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-09-23 11:22:58.020167: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-09-23 11:22:58.020180: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-09-23 11:22:58.020193: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-09-23 11:22:58.020206: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-09-23 11:22:58.020220: I tensorflow/core/common_runtime/bfc_allocator.cc:957] Bin for 2.00MiB was 2.00MiB, Chunk State: 2025-09-23 11:22:58.020232: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 2097152 2025-09-23 11:22:58.020248: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f86b8600000 of size 147456 next 55 2025-09-23 11:22:58.020260: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f86b8624000 of size 65536 next 78 2025-09-23 11:22:58.020272: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f86b8634000 of size 4096 next 190 2025-09-23 11:22:58.020284: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f86b8635000 of size 4096 next 191 2025-09-23 11:22:58.020295: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f86b8636000 of size 4096 next 192 2025-09-23 11:22:58.020307: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f86b8637000 of size 256 next 193 2025-09-23 11:22:58.020327: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f86b8637100 of size 256 next 194 2025-09-23 11:22:58.020374: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f86b8637200 of size 69120 next 72 2025-09-23 11:22:58.020391: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f86b8648000 of size 147456 next 71 2025-09-23 11:22:58.020403: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f86b866c000 of size 131072 next 87 2025-09-23 11:22:58.020415: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f86b868c000 of size 262144 next 103 2025-09-23 11:22:58.020427: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f86b86cc000 of size 262144 next 102 2025-09-23 11:22:58.020439: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f86b870c000 of size 999424 next 18446744073709551615 2025-09-23 11:22:58.020450: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 4194304 2025-09-23 11:22:58.020462: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f86b8800000 of size 589824 next 96 2025-09-23 11:22:58.020474: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f86b8890000 of size 262144 next 117 2025-09-23 11:22:58.020486: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f86b88d0000 of size 262144 next 112 2025-09-23 11:22:58.020498: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f86b8910000 of size 524288 next 111 2025-09-23 11:22:58.020509: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f86b8990000 of size 262144 next 135 2025-09-23 11:22:58.020521: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f86b89d0000 of size 327680 next 124 2025-09-23 11:22:58.020533: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f86b8a20000 of size 589824 next 123 2025-09-23 11:22:58.020545: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f86b8ab0000 of size 262144 next 152 2025-09-23 11:22:58.020556: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f86b8af0000 of size 327680 next 142 2025-09-23 11:22:58.020568: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f86b8b40000 of size 786432 next 18446744073709551615 2025-09-23 11:22:58.020579: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 8388608 2025-09-23 11:22:58.020591: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f86b8c00000 of size 589824 next 160 2025-09-23 11:22:58.020603: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f86b8c90000 of size 589824 next 158 2025-09-23 11:22:58.020614: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f86b8d20000 of size 524288 next 171 2025-09-23 11:22:58.020626: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f86b8da0000 of size 2097152 next 189 2025-09-23 11:22:58.020638: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f86b8fa0000 of size 1048576 next 188 2025-09-23 11:22:58.020650: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f86b90a0000 of size 3538944 next 18446744073709551615 2025-09-23 11:22:58.020661: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 3838720 2025-09-23 11:22:58.020674: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f86b9400000 of size 3838720 next 18446744073709551615 2025-09-23 11:22:58.020685: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 1048576 2025-09-23 11:22:58.020697: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e00000 of size 1280 next 1 2025-09-23 11:22:58.020709: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e00500 of size 256 next 5 2025-09-23 11:22:58.020721: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e00600 of size 256 next 8 2025-09-23 11:22:58.020732: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e00700 of size 256 next 9 2025-09-23 11:22:58.020753: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e00800 of size 256 next 10 2025-09-23 11:22:58.020765: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e00900 of size 256 next 11 2025-09-23 11:22:58.020777: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e00a00 of size 256 next 12 2025-09-23 11:22:58.020789: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e00b00 of size 256 next 13 2025-09-23 11:22:58.020800: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e00c00 of size 256 next 17 2025-09-23 11:22:58.020812: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e00d00 of size 256 next 19 2025-09-23 11:22:58.020823: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e00e00 of size 256 next 20 2025-09-23 11:22:58.020835: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e00f00 of size 256 next 21 2025-09-23 11:22:58.020846: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e01000 of size 256 next 22 2025-09-23 11:22:58.020858: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e01100 of size 256 next 24 2025-09-23 11:22:58.020869: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e01200 of size 256 next 25 2025-09-23 11:22:58.020881: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e01300 of size 256 next 23 2025-09-23 11:22:58.020892: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e01400 of size 256 next 28 2025-09-23 11:22:58.020904: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e01500 of size 256 next 29 2025-09-23 11:22:58.020915: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e01600 of size 256 next 30 2025-09-23 11:22:58.020927: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e01700 of size 256 next 31 2025-09-23 11:22:58.020938: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e01800 of size 256 next 33 2025-09-23 11:22:58.020950: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e01900 of size 256 next 34 2025-09-23 11:22:58.020962: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e01a00 of size 1024 next 32 2025-09-23 11:22:58.020973: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e01e00 of size 1024 next 37 2025-09-23 11:22:58.020985: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e02200 of size 1024 next 38 2025-09-23 11:22:58.020997: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e02600 of size 1024 next 39 2025-09-23 11:22:58.021008: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e02a00 of size 1024 next 40 2025-09-23 11:22:58.021020: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e02e00 of size 1024 next 41 2025-09-23 11:22:58.021031: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e03200 of size 1024 next 43 2025-09-23 11:22:58.021043: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e03600 of size 1024 next 44 2025-09-23 11:22:58.021054: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e03a00 of size 1024 next 45 2025-09-23 11:22:58.021065: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e03e00 of size 1024 next 46 2025-09-23 11:22:58.021077: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e04200 of size 256 next 48 2025-09-23 11:22:58.021089: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e04300 of size 256 next 49 2025-09-23 11:22:58.021100: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e04400 of size 256 next 50 2025-09-23 11:22:58.021112: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e04500 of size 256 next 51 2025-09-23 11:22:58.021123: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e04600 of size 256 next 52 2025-09-23 11:22:58.021143: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e04700 of size 256 next 53 2025-09-23 11:22:58.021155: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e04800 of size 256 next 56 2025-09-23 11:22:58.021167: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e04900 of size 256 next 57 2025-09-23 11:22:58.021178: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e04a00 of size 256 next 58 2025-09-23 11:22:58.021190: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e04b00 of size 256 next 14 2025-09-23 11:22:58.021201: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e04c00 of size 256 next 15 2025-09-23 11:22:58.021212: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e04d00 of size 256 next 16 2025-09-23 11:22:58.021224: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e04e00 of size 1024 next 60 2025-09-23 11:22:58.021235: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e05200 of size 1024 next 61 2025-09-23 11:22:58.021247: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e05600 of size 1024 next 62 2025-09-23 11:22:58.021258: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e05a00 of size 1024 next 63 2025-09-23 11:22:58.021270: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e05e00 of size 1024 next 64 2025-09-23 11:22:58.021281: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e06200 of size 256 next 66 2025-09-23 11:22:58.021293: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e06300 of size 256 next 67 2025-09-23 11:22:58.021304: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e06400 of size 256 next 68 2025-09-23 11:22:58.021316: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e06500 of size 256 next 69 2025-09-23 11:22:58.021327: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e06600 of size 256 next 70 2025-09-23 11:22:58.021338: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e06700 of size 256 next 73 2025-09-23 11:22:58.021350: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e06800 of size 256 next 74 2025-09-23 11:22:58.021361: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e06900 of size 256 next 75 2025-09-23 11:22:58.021373: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e06a00 of size 256 next 76 2025-09-23 11:22:58.021384: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e06b00 of size 256 next 77 2025-09-23 11:22:58.021396: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e06c00 of size 1024 next 79 2025-09-23 11:22:58.021407: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e07000 of size 1024 next 80 2025-09-23 11:22:58.021419: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e07400 of size 1024 next 81 2025-09-23 11:22:58.021430: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e07800 of size 1024 next 82 2025-09-23 11:22:58.021441: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e07c00 of size 1024 next 83 2025-09-23 11:22:58.021453: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e08000 of size 256 next 85 2025-09-23 11:22:58.021464: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e08100 of size 256 next 86 2025-09-23 11:22:58.021476: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e08200 of size 512 next 84 2025-09-23 11:22:58.021488: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e08400 of size 512 next 88 2025-09-23 11:22:58.021499: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e08600 of size 512 next 89 2025-09-23 11:22:58.021519: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e08800 of size 512 next 90 2025-09-23 11:22:58.021531: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e08a00 of size 512 next 91 2025-09-23 11:22:58.021542: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e08c00 of size 256 next 93 2025-09-23 11:22:58.021554: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e08d00 of size 256 next 94 2025-09-23 11:22:58.021565: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e08e00 of size 512 next 92 2025-09-23 11:22:58.021577: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e09000 of size 512 next 97 2025-09-23 11:22:58.021588: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e09200 of size 512 next 98 2025-09-23 11:22:58.021600: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e09400 of size 512 next 99 2025-09-23 11:22:58.021611: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e09600 of size 512 next 2 2025-09-23 11:22:58.021623: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e09800 of size 256 next 3 2025-09-23 11:22:58.021634: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e09900 of size 256 next 4 2025-09-23 11:22:58.021646: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e09a00 of size 16384 next 18 2025-09-23 11:22:58.021657: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e0da00 of size 256 next 100 2025-09-23 11:22:58.021669: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e0db00 of size 256 next 101 2025-09-23 11:22:58.021681: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e0dc00 of size 2048 next 104 2025-09-23 11:22:58.021692: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e0e400 of size 2048 next 105 2025-09-23 11:22:58.021704: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e0ec00 of size 2048 next 106 2025-09-23 11:22:58.021715: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e0f400 of size 2048 next 107 2025-09-23 11:22:58.021726: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e0fc00 of size 2048 next 108 2025-09-23 11:22:58.021738: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e10400 of size 256 next 109 2025-09-23 11:22:58.021749: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e10500 of size 256 next 110 2025-09-23 11:22:58.021761: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e10600 of size 2048 next 113 2025-09-23 11:22:58.021772: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e10e00 of size 2048 next 114 2025-09-23 11:22:58.021784: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e11600 of size 2048 next 115 2025-09-23 11:22:58.021795: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e11e00 of size 2048 next 116 2025-09-23 11:22:58.021807: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e12600 of size 2048 next 6 2025-09-23 11:22:58.021819: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e12e00 of size 37632 next 7 2025-09-23 11:22:58.021830: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e1c100 of size 512 next 118 2025-09-23 11:22:58.021842: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e1c300 of size 512 next 119 2025-09-23 11:22:58.021853: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e1c500 of size 512 next 120 2025-09-23 11:22:58.021865: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e1c700 of size 512 next 121 2025-09-23 11:22:58.021876: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e1c900 of size 512 next 122 2025-09-23 11:22:58.021888: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e1cb00 of size 512 next 125 2025-09-23 11:22:58.021907: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e1cd00 of size 512 next 126 2025-09-23 11:22:58.021919: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e1cf00 of size 512 next 127 2025-09-23 11:22:58.021930: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e1d100 of size 512 next 128 2025-09-23 11:22:58.021942: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e1d300 of size 512 next 129 2025-09-23 11:22:58.021953: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e1d500 of size 2048 next 130 2025-09-23 11:22:58.021965: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e1dd00 of size 2048 next 131 2025-09-23 11:22:58.021976: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e1e500 of size 2048 next 132 2025-09-23 11:22:58.021988: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e1ed00 of size 2048 next 133 2025-09-23 11:22:58.021999: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e1f500 of size 2048 next 134 2025-09-23 11:22:58.022011: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e1fd00 of size 512 next 136 2025-09-23 11:22:58.022022: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e1ff00 of size 512 next 137 2025-09-23 11:22:58.022034: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e20100 of size 512 next 138 2025-09-23 11:22:58.022045: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e20300 of size 512 next 139 2025-09-23 11:22:58.022057: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e20500 of size 512 next 140 2025-09-23 11:22:58.022068: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e20700 of size 512 next 141 2025-09-23 11:22:58.022080: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e20900 of size 512 next 143 2025-09-23 11:22:58.022091: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e20b00 of size 512 next 144 2025-09-23 11:22:58.022102: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e20d00 of size 512 next 145 2025-09-23 11:22:58.022114: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e20f00 of size 512 next 146 2025-09-23 11:22:58.022125: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e21100 of size 2048 next 147 2025-09-23 11:22:58.022137: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e21900 of size 2048 next 148 2025-09-23 11:22:58.022148: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e22100 of size 2048 next 149 2025-09-23 11:22:58.022160: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e22900 of size 2048 next 150 2025-09-23 11:22:58.022171: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e23100 of size 2048 next 151 2025-09-23 11:22:58.022183: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e23900 of size 512 next 153 2025-09-23 11:22:58.022194: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e23b00 of size 512 next 154 2025-09-23 11:22:58.022206: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e23d00 of size 512 next 155 2025-09-23 11:22:58.022217: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e23f00 of size 512 next 156 2025-09-23 11:22:58.022229: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e24100 of size 512 next 157 2025-09-23 11:22:58.022240: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e24300 of size 512 next 161 2025-09-23 11:22:58.022252: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e24500 of size 512 next 162 2025-09-23 11:22:58.022263: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e24700 of size 512 next 163 2025-09-23 11:22:58.022282: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e24900 of size 512 next 164 2025-09-23 11:22:58.022294: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e24b00 of size 512 next 165 2025-09-23 11:22:58.022306: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e24d00 of size 2048 next 166 2025-09-23 11:22:58.022317: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e25500 of size 2048 next 167 2025-09-23 11:22:58.022329: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e25d00 of size 2048 next 168 2025-09-23 11:22:58.022340: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e26500 of size 2048 next 169 2025-09-23 11:22:58.022352: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e26d00 of size 2048 next 170 2025-09-23 11:22:58.022363: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e27500 of size 1024 next 172 2025-09-23 11:22:58.022375: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e27900 of size 1024 next 173 2025-09-23 11:22:58.022386: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e27d00 of size 1024 next 174 2025-09-23 11:22:58.022398: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e28100 of size 1024 next 175 2025-09-23 11:22:58.022409: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e28500 of size 1024 next 176 2025-09-23 11:22:58.022421: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e28900 of size 256 next 178 2025-09-23 11:22:58.022432: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e28a00 of size 256 next 179 2025-09-23 11:22:58.022444: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e28b00 of size 1024 next 177 2025-09-23 11:22:58.022455: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e28f00 of size 1024 next 181 2025-09-23 11:22:58.022467: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e29300 of size 1024 next 182 2025-09-23 11:22:58.022478: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e29700 of size 1024 next 183 2025-09-23 11:22:58.022490: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e29b00 of size 1024 next 184 2025-09-23 11:22:58.022501: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e29f00 of size 256 next 186 2025-09-23 11:22:58.022513: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e2a000 of size 256 next 187 2025-09-23 11:22:58.022524: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e2a100 of size 4096 next 185 2025-09-23 11:22:58.022536: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e2b100 of size 4096 next 42 2025-09-23 11:22:58.022547: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e2c100 of size 65536 next 36 2025-09-23 11:22:58.022559: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e3c100 of size 65536 next 35 2025-09-23 11:22:58.022570: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7f87e2e4c100 of size 98304 next 27 2025-09-23 11:22:58.022582: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e64100 of size 147456 next 26 2025-09-23 11:22:58.022593: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e88100 of size 65536 next 47 2025-09-23 11:22:58.022605: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2e98100 of size 65536 next 59 2025-09-23 11:22:58.022616: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2ea8100 of size 65536 next 65 2025-09-23 11:22:58.022628: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7f87e2eb8100 of size 294656 next 18446744073709551615 2025-09-23 11:22:58.022639: I tensorflow/core/common_runtime/bfc_allocator.cc:995] Summary of in-use Chunks by size: 2025-09-23 11:22:58.022663: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 59 Chunks of size 256 totalling 14.8KiB 2025-09-23 11:22:58.022677: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 40 Chunks of size 512 totalling 20.0KiB 2025-09-23 11:22:58.022690: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 30 Chunks of size 1024 totalling 30.0KiB 2025-09-23 11:22:58.022703: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1280 totalling 1.2KiB 2025-09-23 11:22:58.022715: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 25 Chunks of size 2048 totalling 50.0KiB 2025-09-23 11:22:58.022728: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 5 Chunks of size 4096 totalling 20.0KiB 2025-09-23 11:22:58.022741: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 16384 totalling 16.0KiB 2025-09-23 11:22:58.022755: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 37632 totalling 36.8KiB 2025-09-23 11:22:58.022768: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 6 Chunks of size 65536 totalling 384.0KiB 2025-09-23 11:22:58.022781: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 131072 totalling 128.0KiB 2025-09-23 11:22:58.022794: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 147456 totalling 432.0KiB 2025-09-23 11:22:58.022807: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 4 Chunks of size 262144 totalling 1.00MiB 2025-09-23 11:22:58.022820: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 294656 totalling 287.8KiB 2025-09-23 11:22:58.022833: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 327680 totalling 640.0KiB 2025-09-23 11:22:58.022846: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 524288 totalling 1.00MiB 2025-09-23 11:22:58.022859: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 589824 totalling 1.69MiB 2025-09-23 11:22:58.022872: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 786432 totalling 768.0KiB 2025-09-23 11:22:58.022884: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1048576 totalling 1.00MiB 2025-09-23 11:22:58.022897: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 2097152 totalling 2.00MiB 2025-09-23 11:22:58.022910: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 3538944 totalling 3.38MiB 2025-09-23 11:22:58.022923: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 3838720 totalling 3.66MiB 2025-09-23 11:22:58.022935: I tensorflow/core/common_runtime/bfc_allocator.cc:1002] Sum Total of in-use chunks: 16.49MiB 2025-09-23 11:22:58.022948: I tensorflow/core/common_runtime/bfc_allocator.cc:1004] total_region_allocated_bytes_: 19567360 memory_limit_: 2366177280 available bytes: 2346609920 curr_region_allocation_bytes_: 33554432 2025-09-23 11:22:58.022963: I tensorflow/core/common_runtime/bfc_allocator.cc:1010] Stats: Limit: 2366177280 InUse: 17286400 MaxInUse: 17286656 NumAllocs: 595 MaxAllocSize: 4325376 2025-09-23 11:22:58.022982: W tensorflow/core/common_runtime/bfc_allocator.cc:439] ***_**____**********************x__*********************************xxxxxxx*************xxxxxx****** 2025-09-23 11:22:58.023024: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at cwise_ops_common.h:134 : Resource exhausted: OOM when allocating tensor with shape[1,1,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl3453 thcls : [{'id': 3453, 'mtr_user_id': 31, 'name': 'mask_refus_amiens_050123', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4485, 'photo_desc_type': 5732, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}] thcl {'id': 3453, 'mtr_user_id': 31, 'name': 'mask_refus_amiens_050123', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4485, 'photo_desc_type': 5732, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5732 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5732, 'mask_refus_amiens_050123', 16384, 25088, 'mask_refus_amiens_050123', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 1, 5, 13, 23, 59), datetime.datetime(2023, 1, 5, 13, 23, 59)) {'thcl': {'id': 3453, 'mtr_user_id': 31, 'name': 'mask_refus_amiens_050123', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4485, 'photo_desc_type': 5732, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'], 'list_hashtags_csv': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4485, 'svm_hashtag_type_desc': 5732, 'photo_desc_type': 5732, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_refus_amiens_050123 NUM_CLASSES 9 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 Exception in mask_detect : OOM when allocating tensor with shape[1,1,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Add] we want to redo the detection Using TensorFlow backend. max_time_sub_proc : 3600 erreur pendant la detection Catched exception ! (2013, 'Lost connection to MySQL server during query') Connect or reconnect ! in case -12 caffe_path_current : About to save ! 2 After save, about to update current ! 35.24user 47.95system 1:32.63elapsed 89%CPU (0avgtext+0avgdata 4057184maxresident)k 627880inputs+57320outputs (10313major+5326344minor)pagefaults 0swaps