python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 4234 -l 5 ' -s datou_current_4234 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2627424 load datou : 4234 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5275, 'learn_RUBBIA_REFUS_AMIENS_23', 16384, 25088, 'learn_RUBBIA_REFUS_AMIENS_23', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 23, 14, 19, 39), datetime.datetime(2021, 4, 23, 14, 19, 39)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5561, 'learn_rubbia_refus_2500', 2048, 2048, 'learn_rubbia_refus_2500', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2021, 12, 2, 19, 10, 8), datetime.datetime(2021, 12, 2, 19, 10, 8)) load thcls load THCL from format json or kwargs add thcl : 2847 in CacheModelConfig load THCL from format json or kwargs add thcl : 3237 in CacheModelConfig load pdts add pdt : 5275 in CacheModelConfig add pdt : 5561 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 list_input_json: [] Current got : datou_id : 4234, datou_cur_ids : ['2600267'] with mtr_portfolio_ids : ['20682959'] and first list_photo_ids : [] new path : /proc/2627424/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, brightness, blur_detection, rle_unique_nms_with_priority, crop_condition, thcl, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , BFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 3 ; length of list_pids : 3 ; length of list_args : 3 time to download the photos : 0.5166690349578857 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Thu Feb 20 17:20:31 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10814 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-20 17:20:34.081695: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-20 17:20:34.111203: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-20 17:20:34.113523: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f7b48000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-20 17:20:34.113604: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-20 17:20:34.118180: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-20 17:20:34.266387: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x296416a0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-20 17:20:34.266454: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-20 17:20:34.267870: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-20 17:20:34.268373: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-20 17:20:34.271326: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-20 17:20:34.274325: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-20 17:20:34.274817: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-20 17:20:34.277691: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-20 17:20:34.278957: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-20 17:20:34.284093: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-20 17:20:34.285496: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-20 17:20:34.285577: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-20 17:20:34.286300: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-20 17:20:34.286316: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-20 17:20:34.286325: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-20 17:20:34.287570: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9212 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-20 17:20:34.572564: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-20 17:20:34.572727: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-20 17:20:34.572761: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-20 17:20:34.572790: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-20 17:20:34.572818: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-20 17:20:34.572845: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-20 17:20:34.572871: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-20 17:20:34.572900: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-20 17:20:34.575614: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-20 17:20:34.577587: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-20 17:20:34.577708: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-20 17:20:34.577739: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-20 17:20:34.577768: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-20 17:20:34.577797: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-20 17:20:34.577824: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-20 17:20:34.577851: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-20 17:20:34.577880: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-20 17:20:34.580605: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-20 17:20:34.580681: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-20 17:20:34.580696: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-20 17:20:34.580707: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-20 17:20:34.583886: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9212 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2847 thcls : [{'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}] thcl {'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5275 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5275, 'learn_RUBBIA_REFUS_AMIENS_23', 16384, 25088, 'learn_RUBBIA_REFUS_AMIENS_23', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 23, 14, 19, 39), datetime.datetime(2021, 4, 23, 14, 19, 39)) {'thcl': {'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'], 'list_hashtags_csv': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'svm_hashtag_type_desc': 5275, 'photo_desc_type': 5275, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_RUBBIA_REFUS_AMIENS_23 NUM_CLASSES 9 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_RUBBIA_REFUS_AMIENS_23 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-20 17:20:44.132490: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-20 17:20:44.604394: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-20 17:20:47.256778: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.257761: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.60G (3865470464 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.261060: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3478923264 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.261662: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.92G (3131030784 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.262480: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.62G (2817927680 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.263096: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.36G (2536134912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.263658: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282521344 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.263708: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-20 17:20:47.264353: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.264371: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-20 17:20:47.271798: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.271879: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-20 17:20:47.272459: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.272477: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-20 17:20:47.283215: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.283306: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 466.56MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-20 17:20:47.283925: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.283945: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 466.56MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-20 17:20:47.312345: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.312403: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-20 17:20:47.312984: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.313003: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-20 17:20:47.318688: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.318718: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 243.25MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-20 17:20:47.319306: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.319325: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 243.25MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-20 17:20:47.355949: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.356703: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.359027: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.359743: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.407689: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.408296: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.410245: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.410800: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.440879: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.445072: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.446727: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.447325: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.455029: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.455647: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.457711: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.458298: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.465848: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.466384: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.468154: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.468732: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.499189: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.499801: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.500547: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.501122: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.511548: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.512135: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.532449: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.533047: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.533865: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.534476: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.546848: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.547521: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.548114: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.548875: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.553645: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.554381: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.559917: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.560508: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.573236: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.573843: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.578308: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.578915: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.579760: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.580374: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.581615: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.582600: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.595107: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.595715: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.596307: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.596860: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.597391: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.597922: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.598454: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.599061: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.609872: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.610515: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.617763: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.618381: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.686984: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.687628: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.695988: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.696598: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.720928: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.721859: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.722762: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.723669: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.728397: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.729312: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.730307: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.731217: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.732808: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.732850: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-02-20 17:20:47.743163: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.744069: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.754076: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.754994: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.755882: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.756764: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.757662: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-20 17:20:47.758570: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/learn_RUBBIA_REFUS_AMIENS_23 /data/models_weight/learn_RUBBIA_REFUS_AMIENS_23/mask_model.h5 size_local : 256009536 size in s3 : 256009536 create time local : 2021-08-09 09:43:22 create time in s3 : 2021-08-06 18:54:04 mask_model.h5 already exist and didn't need to update list_images length : 3 NEW PHOTO pour l'instant on ne peut pas sauvegarder la photo dans les tile Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.52422 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 14 time for calcul the mask position with numpy : 0.000324249267578125 nb_pixel_total : 289 time to create 1 rle with old method : 0.0014469623565673828 length of segment : 14 time for calcul the mask position with numpy : 8.916854858398438e-05 nb_pixel_total : 798 time to create 1 rle with old method : 0.0014452934265136719 length of segment : 33 time for calcul the mask position with numpy : 5.364418029785156e-05 nb_pixel_total : 297 time to create 1 rle with old method : 0.0006289482116699219 length of segment : 15 time for calcul the mask position with numpy : 0.0007097721099853516 nb_pixel_total : 27741 time to create 1 rle with old method : 0.03684067726135254 length of segment : 177 time for calcul the mask position with numpy : 7.987022399902344e-05 nb_pixel_total : 414 time to create 1 rle with old method : 0.0006000995635986328 length of segment : 22 time for calcul the mask position with numpy : 7.05718994140625e-05 nb_pixel_total : 393 time to create 1 rle with old method : 0.0006809234619140625 length of segment : 18 time for calcul the mask position with numpy : 0.00010180473327636719 nb_pixel_total : 1796 time to create 1 rle with old method : 0.0022687911987304688 length of segment : 78 time for calcul the mask position with numpy : 0.00011014938354492188 nb_pixel_total : 484 time to create 1 rle with old method : 0.0007097721099853516 length of segment : 26 time for calcul the mask position with numpy : 0.00016021728515625 nb_pixel_total : 1169 time to create 1 rle with old method : 0.0015735626220703125 length of segment : 112 time for calcul the mask position with numpy : 5.459785461425781e-05 nb_pixel_total : 432 time to create 1 rle with old method : 0.0006203651428222656 length of segment : 17 time for calcul the mask position with numpy : 5.316734313964844e-05 nb_pixel_total : 668 time to create 1 rle with old method : 0.0008695125579833984 length of segment : 32 time for calcul the mask position with numpy : 0.0002281665802001953 nb_pixel_total : 10611 time to create 1 rle with old method : 0.016260862350463867 length of segment : 122 time for calcul the mask position with numpy : 6.961822509765625e-05 nb_pixel_total : 1169 time to create 1 rle with old method : 0.0017657279968261719 length of segment : 46 time for calcul the mask position with numpy : 7.009506225585938e-05 nb_pixel_total : 401 time to create 1 rle with old method : 0.0008449554443359375 length of segment : 15 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 9 time for calcul the mask position with numpy : 6.4849853515625e-05 nb_pixel_total : 459 time to create 1 rle with old method : 0.0006568431854248047 length of segment : 28 time for calcul the mask position with numpy : 4.982948303222656e-05 nb_pixel_total : 535 time to create 1 rle with old method : 0.0007886886596679688 length of segment : 22 time for calcul the mask position with numpy : 6.556510925292969e-05 nb_pixel_total : 858 time to create 1 rle with old method : 0.0014362335205078125 length of segment : 33 time for calcul the mask position with numpy : 4.267692565917969e-05 nb_pixel_total : 258 time to create 1 rle with old method : 0.0004723072052001953 length of segment : 12 time for calcul the mask position with numpy : 4.315376281738281e-05 nb_pixel_total : 266 time to create 1 rle with old method : 0.0004069805145263672 length of segment : 25 time for calcul the mask position with numpy : 7.200241088867188e-05 nb_pixel_total : 415 time to create 1 rle with old method : 0.0007650852203369141 length of segment : 15 time for calcul the mask position with numpy : 7.367134094238281e-05 nb_pixel_total : 714 time to create 1 rle with old method : 0.0014164447784423828 length of segment : 30 time for calcul the mask position with numpy : 7.295608520507812e-05 nb_pixel_total : 679 time to create 1 rle with old method : 0.0013165473937988281 length of segment : 41 time for calcul the mask position with numpy : 0.00010275840759277344 nb_pixel_total : 377 time to create 1 rle with old method : 0.0005507469177246094 length of segment : 35 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -122.58672 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 8 time for calcul the mask position with numpy : 9.083747863769531e-05 nb_pixel_total : 1715 time to create 1 rle with old method : 0.002180814743041992 length of segment : 44 time for calcul the mask position with numpy : 6.437301635742188e-05 nb_pixel_total : 971 time to create 1 rle with old method : 0.0011227130889892578 length of segment : 79 time for calcul the mask position with numpy : 4.00543212890625e-05 nb_pixel_total : 172 time to create 1 rle with old method : 0.00029277801513671875 length of segment : 14 time for calcul the mask position with numpy : 4.76837158203125e-05 nb_pixel_total : 392 time to create 1 rle with old method : 0.0005548000335693359 length of segment : 21 time for calcul the mask position with numpy : 6.0558319091796875e-05 nb_pixel_total : 1238 time to create 1 rle with old method : 0.0014948844909667969 length of segment : 73 time for calcul the mask position with numpy : 5.435943603515625e-05 nb_pixel_total : 299 time to create 1 rle with old method : 0.0005083084106445312 length of segment : 24 time for calcul the mask position with numpy : 4.8160552978515625e-05 nb_pixel_total : 253 time to create 1 rle with old method : 0.0004336833953857422 length of segment : 18 time for calcul the mask position with numpy : 0.000110626220703125 nb_pixel_total : 3616 time to create 1 rle with old method : 0.0043828487396240234 length of segment : 67 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.23906 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 18 time for calcul the mask position with numpy : 8.106231689453125e-05 nb_pixel_total : 249 time to create 1 rle with old method : 0.00046634674072265625 length of segment : 22 time for calcul the mask position with numpy : 6.103515625e-05 nb_pixel_total : 1366 time to create 1 rle with old method : 0.0016448497772216797 length of segment : 58 time for calcul the mask position with numpy : 5.698204040527344e-05 nb_pixel_total : 704 time to create 1 rle with old method : 0.0009827613830566406 length of segment : 30 time for calcul the mask position with numpy : 4.9114227294921875e-05 nb_pixel_total : 278 time to create 1 rle with old method : 0.00040411949157714844 length of segment : 21 time for calcul the mask position with numpy : 3.4809112548828125e-05 nb_pixel_total : 145 time to create 1 rle with old method : 0.00025391578674316406 length of segment : 9 time for calcul the mask position with numpy : 3.838539123535156e-05 nb_pixel_total : 155 time to create 1 rle with old method : 0.0002582073211669922 length of segment : 12 time for calcul the mask position with numpy : 4.57763671875e-05 nb_pixel_total : 162 time to create 1 rle with old method : 0.0003974437713623047 length of segment : 13 time for calcul the mask position with numpy : 9.489059448242188e-05 nb_pixel_total : 2098 time to create 1 rle with old method : 0.002567768096923828 length of segment : 46 time for calcul the mask position with numpy : 7.867813110351562e-05 nb_pixel_total : 1642 time to create 1 rle with old method : 0.0024683475494384766 length of segment : 37 time for calcul the mask position with numpy : 0.0001385211944580078 nb_pixel_total : 5413 time to create 1 rle with old method : 0.006575107574462891 length of segment : 89 time for calcul the mask position with numpy : 0.00011897087097167969 nb_pixel_total : 3512 time to create 1 rle with old method : 0.004517555236816406 length of segment : 75 time for calcul the mask position with numpy : 7.891654968261719e-05 nb_pixel_total : 1992 time to create 1 rle with old method : 0.0027260780334472656 length of segment : 40 time for calcul the mask position with numpy : 6.747245788574219e-05 nb_pixel_total : 933 time to create 1 rle with old method : 0.0014243125915527344 length of segment : 50 time for calcul the mask position with numpy : 4.696846008300781e-05 nb_pixel_total : 358 time to create 1 rle with old method : 0.00048804283142089844 length of segment : 25 time for calcul the mask position with numpy : 5.340576171875e-05 nb_pixel_total : 390 time to create 1 rle with old method : 0.0006465911865234375 length of segment : 42 time for calcul the mask position with numpy : 5.5789947509765625e-05 nb_pixel_total : 338 time to create 1 rle with old method : 0.0005488395690917969 length of segment : 19 time for calcul the mask position with numpy : 0.00010156631469726562 nb_pixel_total : 1362 time to create 1 rle with old method : 0.0017480850219726562 length of segment : 49 time for calcul the mask position with numpy : 0.000141143798828125 nb_pixel_total : 4048 time to create 1 rle with old method : 0.0046842098236083984 length of segment : 70 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -119.34063 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 14 time for calcul the mask position with numpy : 8.344650268554688e-05 nb_pixel_total : 1096 time to create 1 rle with old method : 0.0013194084167480469 length of segment : 55 time for calcul the mask position with numpy : 7.414817810058594e-05 nb_pixel_total : 807 time to create 1 rle with old method : 0.0011358261108398438 length of segment : 51 time for calcul the mask position with numpy : 4.649162292480469e-05 nb_pixel_total : 588 time to create 1 rle with old method : 0.0007965564727783203 length of segment : 22 time for calcul the mask position with numpy : 0.00016450881958007812 nb_pixel_total : 6643 time to create 1 rle with old method : 0.007844924926757812 length of segment : 98 time for calcul the mask position with numpy : 8.344650268554688e-05 nb_pixel_total : 136 time to create 1 rle with old method : 0.00023889541625976562 length of segment : 14 time for calcul the mask position with numpy : 5.507469177246094e-05 nb_pixel_total : 249 time to create 1 rle with old method : 0.0005550384521484375 length of segment : 14 time for calcul the mask position with numpy : 6.937980651855469e-05 nb_pixel_total : 274 time to create 1 rle with old method : 0.0006077289581298828 length of segment : 18 time for calcul the mask position with numpy : 6.961822509765625e-05 nb_pixel_total : 562 time to create 1 rle with old method : 0.00116729736328125 length of segment : 23 time for calcul the mask position with numpy : 8.511543273925781e-05 nb_pixel_total : 326 time to create 1 rle with old method : 0.0006954669952392578 length of segment : 17 time for calcul the mask position with numpy : 0.000202178955078125 nb_pixel_total : 5159 time to create 1 rle with old method : 0.006055355072021484 length of segment : 151 time for calcul the mask position with numpy : 9.560585021972656e-05 nb_pixel_total : 1922 time to create 1 rle with old method : 0.0032262802124023438 length of segment : 55 time for calcul the mask position with numpy : 5.888938903808594e-05 nb_pixel_total : 255 time to create 1 rle with old method : 0.0005407333374023438 length of segment : 25 time for calcul the mask position with numpy : 9.822845458984375e-05 nb_pixel_total : 2182 time to create 1 rle with old method : 0.003700733184814453 length of segment : 51 time for calcul the mask position with numpy : 7.772445678710938e-05 nb_pixel_total : 810 time to create 1 rle with old method : 0.0014758110046386719 length of segment : 33 Processing 1 images image shape: (400, 400, 3) min: 6.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -110.52031 max: 146.78359 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 5 time for calcul the mask position with numpy : 0.00019693374633789062 nb_pixel_total : 6019 time to create 1 rle with old method : 0.01030421257019043 length of segment : 92 time for calcul the mask position with numpy : 0.00014257431030273438 nb_pixel_total : 5567 time to create 1 rle with old method : 0.00661921501159668 length of segment : 74 time for calcul the mask position with numpy : 0.00010442733764648438 nb_pixel_total : 496 time to create 1 rle with old method : 0.0010805130004882812 length of segment : 21 time for calcul the mask position with numpy : 8.106231689453125e-05 nb_pixel_total : 629 time to create 1 rle with old method : 0.0013170242309570312 length of segment : 38 time for calcul the mask position with numpy : 9.202957153320312e-05 nb_pixel_total : 721 time to create 1 rle with old method : 0.001451730728149414 length of segment : 48 Processing 1 images image shape: (400, 160, 3) min: 1.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 145.16641 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 3 time for calcul the mask position with numpy : 0.00011610984802246094 nb_pixel_total : 2757 time to create 1 rle with old method : 0.00529789924621582 length of segment : 94 time for calcul the mask position with numpy : 0.00028395652770996094 nb_pixel_total : 3428 time to create 1 rle with old method : 0.006517171859741211 length of segment : 210 time for calcul the mask position with numpy : 0.00031304359436035156 nb_pixel_total : 9384 time to create 1 rle with old method : 0.016966581344604492 length of segment : 275 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -119.17656 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 16 time for calcul the mask position with numpy : 7.653236389160156e-05 nb_pixel_total : 414 time to create 1 rle with old method : 0.0008056163787841797 length of segment : 33 time for calcul the mask position with numpy : 5.0067901611328125e-05 nb_pixel_total : 310 time to create 1 rle with old method : 0.0006215572357177734 length of segment : 25 time for calcul the mask position with numpy : 4.315376281738281e-05 nb_pixel_total : 149 time to create 1 rle with old method : 0.00035953521728515625 length of segment : 17 time for calcul the mask position with numpy : 4.76837158203125e-05 nb_pixel_total : 368 time to create 1 rle with old method : 0.0007281303405761719 length of segment : 29 time for calcul the mask position with numpy : 6.556510925292969e-05 nb_pixel_total : 831 time to create 1 rle with old method : 0.0015933513641357422 length of segment : 63 time for calcul the mask position with numpy : 5.030632019042969e-05 nb_pixel_total : 213 time to create 1 rle with old method : 0.00043773651123046875 length of segment : 19 time for calcul the mask position with numpy : 9.799003601074219e-05 nb_pixel_total : 1723 time to create 1 rle with old method : 0.0030777454376220703 length of segment : 103 time for calcul the mask position with numpy : 4.458427429199219e-05 nb_pixel_total : 92 time to create 1 rle with old method : 0.00016427040100097656 length of segment : 27 time for calcul the mask position with numpy : 3.910064697265625e-05 nb_pixel_total : 337 time to create 1 rle with old method : 0.00047278404235839844 length of segment : 17 time for calcul the mask position with numpy : 4.267692565917969e-05 nb_pixel_total : 68 time to create 1 rle with old method : 0.0001614093780517578 length of segment : 21 time for calcul the mask position with numpy : 4.553794860839844e-05 nb_pixel_total : 676 time to create 1 rle with old method : 0.000827789306640625 length of segment : 39 time for calcul the mask position with numpy : 5.221366882324219e-05 nb_pixel_total : 431 time to create 1 rle with old method : 0.0006155967712402344 length of segment : 16 time for calcul the mask position with numpy : 3.6716461181640625e-05 nb_pixel_total : 242 time to create 1 rle with old method : 0.0003352165222167969 length of segment : 28 time for calcul the mask position with numpy : 3.814697265625e-05 nb_pixel_total : 396 time to create 1 rle with old method : 0.0006198883056640625 length of segment : 25 time for calcul the mask position with numpy : 3.457069396972656e-05 nb_pixel_total : 177 time to create 1 rle with old method : 0.0002505779266357422 length of segment : 24 time for calcul the mask position with numpy : 3.123283386230469e-05 nb_pixel_total : 167 time to create 1 rle with old method : 0.00024580955505371094 length of segment : 24 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -122.55156 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 18 time for calcul the mask position with numpy : 7.390975952148438e-05 nb_pixel_total : 210 time to create 1 rle with old method : 0.0003650188446044922 length of segment : 21 time for calcul the mask position with numpy : 7.152557373046875e-05 nb_pixel_total : 1342 time to create 1 rle with old method : 0.0016522407531738281 length of segment : 65 time for calcul the mask position with numpy : 5.936622619628906e-05 nb_pixel_total : 812 time to create 1 rle with old method : 0.0010423660278320312 length of segment : 39 time for calcul the mask position with numpy : 5.841255187988281e-05 nb_pixel_total : 965 time to create 1 rle with old method : 0.0012764930725097656 length of segment : 33 time for calcul the mask position with numpy : 5.984306335449219e-05 nb_pixel_total : 575 time to create 1 rle with old method : 0.0008234977722167969 length of segment : 48 time for calcul the mask position with numpy : 3.4809112548828125e-05 nb_pixel_total : 59 time to create 1 rle with old method : 0.00011348724365234375 length of segment : 11 time for calcul the mask position with numpy : 6.413459777832031e-05 nb_pixel_total : 1291 time to create 1 rle with old method : 0.0015189647674560547 length of segment : 104 time for calcul the mask position with numpy : 0.00011563301086425781 nb_pixel_total : 2993 time to create 1 rle with old method : 0.00347900390625 length of segment : 96 time for calcul the mask position with numpy : 9.942054748535156e-05 nb_pixel_total : 2524 time to create 1 rle with old method : 0.003208637237548828 length of segment : 84 time for calcul the mask position with numpy : 3.743171691894531e-05 nb_pixel_total : 63 time to create 1 rle with old method : 0.00011157989501953125 length of segment : 13 time for calcul the mask position with numpy : 4.601478576660156e-05 nb_pixel_total : 819 time to create 1 rle with old method : 0.001079559326171875 length of segment : 31 time for calcul the mask position with numpy : 0.00010442733764648438 nb_pixel_total : 2445 time to create 1 rle with old method : 0.003023862838745117 length of segment : 78 time for calcul the mask position with numpy : 3.743171691894531e-05 nb_pixel_total : 215 time to create 1 rle with old method : 0.00032401084899902344 length of segment : 15 time for calcul the mask position with numpy : 3.528594970703125e-05 nb_pixel_total : 187 time to create 1 rle with old method : 0.00027370452880859375 length of segment : 18 time for calcul the mask position with numpy : 4.5299530029296875e-05 nb_pixel_total : 779 time to create 1 rle with old method : 0.001016855239868164 length of segment : 57 time for calcul the mask position with numpy : 8.7738037109375e-05 nb_pixel_total : 962 time to create 1 rle with old method : 0.0020787715911865234 length of segment : 36 time for calcul the mask position with numpy : 5.5789947509765625e-05 nb_pixel_total : 539 time to create 1 rle with old method : 0.0006492137908935547 length of segment : 47 time for calcul the mask position with numpy : 4.267692565917969e-05 nb_pixel_total : 520 time to create 1 rle with old method : 0.000667572021484375 length of segment : 49 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -121.37578 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 24 time for calcul the mask position with numpy : 6.67572021484375e-05 nb_pixel_total : 425 time to create 1 rle with old method : 0.0005645751953125 length of segment : 25 time for calcul the mask position with numpy : 3.9577484130859375e-05 nb_pixel_total : 92 time to create 1 rle with old method : 0.0001678466796875 length of segment : 9 time for calcul the mask position with numpy : 3.814697265625e-05 nb_pixel_total : 411 time to create 1 rle with old method : 0.0005404949188232422 length of segment : 24 time for calcul the mask position with numpy : 4.076957702636719e-05 nb_pixel_total : 324 time to create 1 rle with old method : 0.0004782676696777344 length of segment : 25 time for calcul the mask position with numpy : 4.8160552978515625e-05 nb_pixel_total : 855 time to create 1 rle with old method : 0.0010673999786376953 length of segment : 34 time for calcul the mask position with numpy : 4.839897155761719e-05 nb_pixel_total : 900 time to create 1 rle with old method : 0.0011818408966064453 length of segment : 37 time for calcul the mask position with numpy : 6.508827209472656e-05 nb_pixel_total : 1012 time to create 1 rle with old method : 0.001245737075805664 length of segment : 79 time for calcul the mask position with numpy : 3.5762786865234375e-05 nb_pixel_total : 148 time to create 1 rle with old method : 0.0002467632293701172 length of segment : 12 time for calcul the mask position with numpy : 3.910064697265625e-05 nb_pixel_total : 406 time to create 1 rle with old method : 0.0005459785461425781 length of segment : 29 time for calcul the mask position with numpy : 4.5299530029296875e-05 nb_pixel_total : 296 time to create 1 rle with old method : 0.00041413307189941406 length of segment : 26 time for calcul the mask position with numpy : 7.295608520507812e-05 nb_pixel_total : 2003 time to create 1 rle with old method : 0.002553224563598633 length of segment : 42 time for calcul the mask position with numpy : 5.1021575927734375e-05 nb_pixel_total : 1262 time to create 1 rle with old method : 0.001672506332397461 length of segment : 31 time for calcul the mask position with numpy : 4.100799560546875e-05 nb_pixel_total : 429 time to create 1 rle with old method : 0.0005574226379394531 length of segment : 29 time for calcul the mask position with numpy : 4.100799560546875e-05 nb_pixel_total : 525 time to create 1 rle with old method : 0.0007104873657226562 length of segment : 32 time for calcul the mask position with numpy : 8.845329284667969e-05 nb_pixel_total : 1407 time to create 1 rle with old method : 0.0016629695892333984 length of segment : 115 time for calcul the mask position with numpy : 5.269050598144531e-05 nb_pixel_total : 299 time to create 1 rle with old method : 0.0005395412445068359 length of segment : 23 time for calcul the mask position with numpy : 7.081031799316406e-05 nb_pixel_total : 1157 time to create 1 rle with old method : 0.0015189647674560547 length of segment : 36 time for calcul the mask position with numpy : 0.0001938343048095703 nb_pixel_total : 6199 time to create 1 rle with old method : 0.0072498321533203125 length of segment : 104 time for calcul the mask position with numpy : 4.696846008300781e-05 nb_pixel_total : 150 time to create 1 rle with old method : 0.00022673606872558594 length of segment : 16 time for calcul the mask position with numpy : 6.365776062011719e-05 nb_pixel_total : 1174 time to create 1 rle with old method : 0.001489400863647461 length of segment : 43 time for calcul the mask position with numpy : 0.00015044212341308594 nb_pixel_total : 6005 time to create 1 rle with old method : 0.006830453872680664 length of segment : 140 time for calcul the mask position with numpy : 4.6253204345703125e-05 nb_pixel_total : 274 time to create 1 rle with old method : 0.00038909912109375 length of segment : 25 time for calcul the mask position with numpy : 7.724761962890625e-05 nb_pixel_total : 845 time to create 1 rle with old method : 0.0010514259338378906 length of segment : 67 time for calcul the mask position with numpy : 0.00015497207641601562 nb_pixel_total : 3792 time to create 1 rle with old method : 0.004424095153808594 length of segment : 108 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.67656 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 36 time for calcul the mask position with numpy : 0.00020575523376464844 nb_pixel_total : 2404 time to create 1 rle with old method : 0.003314971923828125 length of segment : 60 time for calcul the mask position with numpy : 6.318092346191406e-05 nb_pixel_total : 388 time to create 1 rle with old method : 0.000545501708984375 length of segment : 27 time for calcul the mask position with numpy : 6.961822509765625e-05 nb_pixel_total : 435 time to create 1 rle with old method : 0.0006906986236572266 length of segment : 34 time for calcul the mask position with numpy : 8.559226989746094e-05 nb_pixel_total : 1408 time to create 1 rle with old method : 0.0018227100372314453 length of segment : 47 time for calcul the mask position with numpy : 7.653236389160156e-05 nb_pixel_total : 1162 time to create 1 rle with old method : 0.0014951229095458984 length of segment : 52 time for calcul the mask position with numpy : 7.677078247070312e-05 nb_pixel_total : 391 time to create 1 rle with old method : 0.0008187294006347656 length of segment : 22 time for calcul the mask position with numpy : 8.034706115722656e-05 nb_pixel_total : 293 time to create 1 rle with old method : 0.0006163120269775391 length of segment : 23 time for calcul the mask position with numpy : 7.319450378417969e-05 nb_pixel_total : 483 time to create 1 rle with old method : 0.0009765625 length of segment : 28 time for calcul the mask position with numpy : 9.72747802734375e-05 nb_pixel_total : 780 time to create 1 rle with old method : 0.0015094280242919922 length of segment : 49 time for calcul the mask position with numpy : 5.6743621826171875e-05 nb_pixel_total : 370 time to create 1 rle with old method : 0.0005159378051757812 length of segment : 31 time for calcul the mask position with numpy : 6.365776062011719e-05 nb_pixel_total : 557 time to create 1 rle with old method : 0.0007367134094238281 length of segment : 54 time for calcul the mask position with numpy : 5.0067901611328125e-05 nb_pixel_total : 183 time to create 1 rle with old method : 0.0002989768981933594 length of segment : 27 time for calcul the mask position with numpy : 4.410743713378906e-05 nb_pixel_total : 188 time to create 1 rle with old method : 0.0002841949462890625 length of segment : 24 time for calcul the mask position with numpy : 4.792213439941406e-05 nb_pixel_total : 408 time to create 1 rle with old method : 0.0006077289581298828 length of segment : 24 time for calcul the mask position with numpy : 5.698204040527344e-05 nb_pixel_total : 915 time to create 1 rle with old method : 0.0012238025665283203 length of segment : 40 time for calcul the mask position with numpy : 8.034706115722656e-05 nb_pixel_total : 607 time to create 1 rle with old method : 0.0008323192596435547 length of segment : 99 time for calcul the mask position with numpy : 0.00011420249938964844 nb_pixel_total : 2406 time to create 1 rle with old method : 0.00292205810546875 length of segment : 80 time for calcul the mask position with numpy : 0.0002410411834716797 nb_pixel_total : 6520 time to create 1 rle with old method : 0.00855565071105957 length of segment : 107 time for calcul the mask position with numpy : 6.985664367675781e-05 nb_pixel_total : 636 time to create 1 rle with old method : 0.0008785724639892578 length of segment : 41 time for calcul the mask position with numpy : 0.00011277198791503906 nb_pixel_total : 2260 time to create 1 rle with old method : 0.0029506683349609375 length of segment : 67 time for calcul the mask position with numpy : 5.245208740234375e-05 nb_pixel_total : 359 time to create 1 rle with old method : 0.0004966259002685547 length of segment : 21 time for calcul the mask position with numpy : 6.246566772460938e-05 nb_pixel_total : 901 time to create 1 rle with old method : 0.0011928081512451172 length of segment : 37 time for calcul the mask position with numpy : 6.771087646484375e-05 nb_pixel_total : 557 time to create 1 rle with old method : 0.0006804466247558594 length of segment : 58 time for calcul the mask position with numpy : 6.556510925292969e-05 nb_pixel_total : 1108 time to create 1 rle with old method : 0.001409292221069336 length of segment : 46 time for calcul the mask position with numpy : 0.0001888275146484375 nb_pixel_total : 4846 time to create 1 rle with old method : 0.005920886993408203 length of segment : 145 time for calcul the mask position with numpy : 9.34600830078125e-05 nb_pixel_total : 977 time to create 1 rle with old method : 0.001888275146484375 length of segment : 38 time for calcul the mask position with numpy : 7.343292236328125e-05 nb_pixel_total : 493 time to create 1 rle with old method : 0.0007112026214599609 length of segment : 28 time for calcul the mask position with numpy : 7.343292236328125e-05 nb_pixel_total : 291 time to create 1 rle with old method : 0.000492095947265625 length of segment : 27 time for calcul the mask position with numpy : 5.5789947509765625e-05 nb_pixel_total : 363 time to create 1 rle with old method : 0.000545501708984375 length of segment : 21 time for calcul the mask position with numpy : 4.00543212890625e-05 nb_pixel_total : 277 time to create 1 rle with old method : 0.00037550926208496094 length of segment : 25 time for calcul the mask position with numpy : 8.463859558105469e-05 nb_pixel_total : 2555 time to create 1 rle with old method : 0.003191709518432617 length of segment : 74 time for calcul the mask position with numpy : 0.00014638900756835938 nb_pixel_total : 3817 time to create 1 rle with old method : 0.004929065704345703 length of segment : 79 time for calcul the mask position with numpy : 5.340576171875e-05 nb_pixel_total : 371 time to create 1 rle with old method : 0.0005121231079101562 length of segment : 32 time for calcul the mask position with numpy : 5.745887756347656e-05 nb_pixel_total : 956 time to create 1 rle with old method : 0.001237630844116211 length of segment : 37 time for calcul the mask position with numpy : 0.00017595291137695312 nb_pixel_total : 6411 time to create 1 rle with old method : 0.008503437042236328 length of segment : 106 time for calcul the mask position with numpy : 5.173683166503906e-05 nb_pixel_total : 217 time to create 1 rle with old method : 0.00048470497131347656 length of segment : 17 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 27 time for calcul the mask position with numpy : 8.749961853027344e-05 nb_pixel_total : 2045 time to create 1 rle with old method : 0.0025022029876708984 length of segment : 48 time for calcul the mask position with numpy : 8.20159912109375e-05 nb_pixel_total : 1493 time to create 1 rle with old method : 0.0020346641540527344 length of segment : 66 time for calcul the mask position with numpy : 0.00010538101196289062 nb_pixel_total : 1798 time to create 1 rle with old method : 0.0035076141357421875 length of segment : 53 time for calcul the mask position with numpy : 7.414817810058594e-05 nb_pixel_total : 723 time to create 1 rle with old method : 0.0011699199676513672 length of segment : 24 time for calcul the mask position with numpy : 7.963180541992188e-05 nb_pixel_total : 900 time to create 1 rle with old method : 0.0014617443084716797 length of segment : 35 time for calcul the mask position with numpy : 7.796287536621094e-05 nb_pixel_total : 477 time to create 1 rle with old method : 0.00084686279296875 length of segment : 38 time for calcul the mask position with numpy : 7.963180541992188e-05 nb_pixel_total : 1042 time to create 1 rle with old method : 0.0014567375183105469 length of segment : 38 time for calcul the mask position with numpy : 7.987022399902344e-05 nb_pixel_total : 521 time to create 1 rle with old method : 0.0008754730224609375 length of segment : 56 time for calcul the mask position with numpy : 9.489059448242188e-05 nb_pixel_total : 1302 time to create 1 rle with old method : 0.0022454261779785156 length of segment : 51 time for calcul the mask position with numpy : 0.00014448165893554688 nb_pixel_total : 3682 time to create 1 rle with old method : 0.007655143737792969 length of segment : 97 time for calcul the mask position with numpy : 0.00015687942504882812 nb_pixel_total : 1924 time to create 1 rle with old method : 0.0027723312377929688 length of segment : 69 time for calcul the mask position with numpy : 0.00013685226440429688 nb_pixel_total : 913 time to create 1 rle with old method : 0.002777099609375 length of segment : 30 time for calcul the mask position with numpy : 5.936622619628906e-05 nb_pixel_total : 236 time to create 1 rle with old method : 0.0004761219024658203 length of segment : 17 time for calcul the mask position with numpy : 5.1975250244140625e-05 nb_pixel_total : 585 time to create 1 rle with old method : 0.000858306884765625 length of segment : 34 time for calcul the mask position with numpy : 6.937980651855469e-05 nb_pixel_total : 1293 time to create 1 rle with old method : 0.0016257762908935547 length of segment : 33 time for calcul the mask position with numpy : 6.866455078125e-05 nb_pixel_total : 1213 time to create 1 rle with old method : 0.0015521049499511719 length of segment : 43 time for calcul the mask position with numpy : 8.654594421386719e-05 nb_pixel_total : 623 time to create 1 rle with old method : 0.0010554790496826172 length of segment : 30 time for calcul the mask position with numpy : 4.38690185546875e-05 nb_pixel_total : 216 time to create 1 rle with old method : 0.00034880638122558594 length of segment : 27 time for calcul the mask position with numpy : 5.7220458984375e-05 nb_pixel_total : 487 time to create 1 rle with old method : 0.0009243488311767578 length of segment : 35 time for calcul the mask position with numpy : 5.054473876953125e-05 nb_pixel_total : 399 time to create 1 rle with old method : 0.0005593299865722656 length of segment : 34 time for calcul the mask position with numpy : 0.00010704994201660156 nb_pixel_total : 1899 time to create 1 rle with old method : 0.0029206275939941406 length of segment : 143 time for calcul the mask position with numpy : 6.318092346191406e-05 nb_pixel_total : 346 time to create 1 rle with old method : 0.0006542205810546875 length of segment : 19 time for calcul the mask position with numpy : 7.104873657226562e-05 nb_pixel_total : 271 time to create 1 rle with old method : 0.0006220340728759766 length of segment : 20 time for calcul the mask position with numpy : 0.00010585784912109375 nb_pixel_total : 2276 time to create 1 rle with old method : 0.0033445358276367188 length of segment : 54 time for calcul the mask position with numpy : 0.00010585784912109375 nb_pixel_total : 1459 time to create 1 rle with old method : 0.0019659996032714844 length of segment : 47 time for calcul the mask position with numpy : 6.079673767089844e-05 nb_pixel_total : 988 time to create 1 rle with old method : 0.0012240409851074219 length of segment : 39 time for calcul the mask position with numpy : 5.626678466796875e-05 nb_pixel_total : 692 time to create 1 rle with old method : 0.0009601116180419922 length of segment : 30 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -121.41484 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 3 time for calcul the mask position with numpy : 5.459785461425781e-05 nb_pixel_total : 690 time to create 1 rle with old method : 0.001027822494506836 length of segment : 39 time for calcul the mask position with numpy : 5.817413330078125e-05 nb_pixel_total : 372 time to create 1 rle with old method : 0.0005977153778076172 length of segment : 31 time for calcul the mask position with numpy : 5.2928924560546875e-05 nb_pixel_total : 745 time to create 1 rle with old method : 0.000978708267211914 length of segment : 36 Processing 1 images image shape: (400, 160, 3) min: 0.00000 max: 176.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 67.69766 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 2 time for calcul the mask position with numpy : 7.700920104980469e-05 nb_pixel_total : 1349 time to create 1 rle with old method : 0.001844167709350586 length of segment : 75 time for calcul the mask position with numpy : 8.392333984375e-05 nb_pixel_total : 1263 time to create 1 rle with old method : 0.001827239990234375 length of segment : 88 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.45391 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 11 time for calcul the mask position with numpy : 4.792213439941406e-05 nb_pixel_total : 194 time to create 1 rle with old method : 0.00029969215393066406 length of segment : 16 time for calcul the mask position with numpy : 4.6253204345703125e-05 nb_pixel_total : 618 time to create 1 rle with old method : 0.0008091926574707031 length of segment : 43 time for calcul the mask position with numpy : 6.103515625e-05 nb_pixel_total : 1639 time to create 1 rle with old method : 0.0020351409912109375 length of segment : 56 time for calcul the mask position with numpy : 8.225440979003906e-05 nb_pixel_total : 233 time to create 1 rle with old method : 0.0005097389221191406 length of segment : 33 time for calcul the mask position with numpy : 6.198883056640625e-05 nb_pixel_total : 142 time to create 1 rle with old method : 0.0003082752227783203 length of segment : 13 time for calcul the mask position with numpy : 6.508827209472656e-05 nb_pixel_total : 171 time to create 1 rle with old method : 0.00027060508728027344 length of segment : 29 time for calcul the mask position with numpy : 5.817413330078125e-05 nb_pixel_total : 765 time to create 1 rle with old method : 0.0010161399841308594 length of segment : 37 time for calcul the mask position with numpy : 5.14984130859375e-05 nb_pixel_total : 326 time to create 1 rle with old method : 0.00044989585876464844 length of segment : 28 time for calcul the mask position with numpy : 6.532669067382812e-05 nb_pixel_total : 380 time to create 1 rle with old method : 0.0008211135864257812 length of segment : 18 time for calcul the mask position with numpy : 9.226799011230469e-05 nb_pixel_total : 1472 time to create 1 rle with old method : 0.0017864704132080078 length of segment : 85 time for calcul the mask position with numpy : 5.340576171875e-05 nb_pixel_total : 698 time to create 1 rle with old method : 0.0008938312530517578 length of segment : 48 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.05938 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 20 time for calcul the mask position with numpy : 6.461143493652344e-05 nb_pixel_total : 454 time to create 1 rle with old method : 0.0006077289581298828 length of segment : 23 time for calcul the mask position with numpy : 5.459785461425781e-05 nb_pixel_total : 623 time to create 1 rle with old method : 0.0008189678192138672 length of segment : 56 time for calcul the mask position with numpy : 5.340576171875e-05 nb_pixel_total : 466 time to create 1 rle with old method : 0.0006265640258789062 length of segment : 30 time for calcul the mask position with numpy : 7.963180541992188e-05 nb_pixel_total : 2370 time to create 1 rle with old method : 0.0027923583984375 length of segment : 58 time for calcul the mask position with numpy : 5.984306335449219e-05 nb_pixel_total : 239 time to create 1 rle with old method : 0.0003871917724609375 length of segment : 15 time for calcul the mask position with numpy : 0.00010061264038085938 nb_pixel_total : 2269 time to create 1 rle with old method : 0.0027778148651123047 length of segment : 59 time for calcul the mask position with numpy : 5.507469177246094e-05 nb_pixel_total : 93 time to create 1 rle with old method : 0.0001747608184814453 length of segment : 9 time for calcul the mask position with numpy : 6.437301635742188e-05 nb_pixel_total : 268 time to create 1 rle with old method : 0.0003764629364013672 length of segment : 31 time for calcul the mask position with numpy : 0.0001246929168701172 nb_pixel_total : 2982 time to create 1 rle with old method : 0.005354166030883789 length of segment : 63 time for calcul the mask position with numpy : 4.363059997558594e-05 nb_pixel_total : 179 time to create 1 rle with old method : 0.00027441978454589844 length of segment : 15 time for calcul the mask position with numpy : 8.606910705566406e-05 nb_pixel_total : 2964 time to create 1 rle with old method : 0.004482746124267578 length of segment : 77 time for calcul the mask position with numpy : 0.00014781951904296875 nb_pixel_total : 4948 time to create 1 rle with old method : 0.005648374557495117 length of segment : 106 time for calcul the mask position with numpy : 0.0001068115234375 nb_pixel_total : 1689 time to create 1 rle with old method : 0.0025174617767333984 length of segment : 68 time for calcul the mask position with numpy : 8.845329284667969e-05 nb_pixel_total : 2761 time to create 1 rle with old method : 0.004673957824707031 length of segment : 69 time for calcul the mask position with numpy : 5.1975250244140625e-05 nb_pixel_total : 568 time to create 1 rle with old method : 0.0007302761077880859 length of segment : 35 time for calcul the mask position with numpy : 3.0517578125e-05 nb_pixel_total : 110 time to create 1 rle with old method : 0.00017547607421875 length of segment : 12 time for calcul the mask position with numpy : 4.4345855712890625e-05 nb_pixel_total : 794 time to create 1 rle with old method : 0.0010333061218261719 length of segment : 41 time for calcul the mask position with numpy : 3.695487976074219e-05 nb_pixel_total : 320 time to create 1 rle with old method : 0.0004138946533203125 length of segment : 29 time for calcul the mask position with numpy : 0.0001266002655029297 nb_pixel_total : 4894 time to create 1 rle with old method : 0.0054931640625 length of segment : 106 time for calcul the mask position with numpy : 4.00543212890625e-05 nb_pixel_total : 309 time to create 1 rle with old method : 0.0004200935363769531 length of segment : 28 Processing 1 images image shape: (400, 400, 3) min: 7.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -111.84844 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 26 time for calcul the mask position with numpy : 7.653236389160156e-05 nb_pixel_total : 293 time to create 1 rle with old method : 0.0005314350128173828 length of segment : 28 time for calcul the mask position with numpy : 7.772445678710938e-05 nb_pixel_total : 716 time to create 1 rle with old method : 0.0014393329620361328 length of segment : 41 time for calcul the mask position with numpy : 6.413459777832031e-05 nb_pixel_total : 277 time to create 1 rle with old method : 0.0006053447723388672 length of segment : 27 time for calcul the mask position with numpy : 6.127357482910156e-05 nb_pixel_total : 222 time to create 1 rle with old method : 0.0004963874816894531 length of segment : 20 time for calcul the mask position with numpy : 0.00013303756713867188 nb_pixel_total : 3654 time to create 1 rle with old method : 0.005997896194458008 length of segment : 134 time for calcul the mask position with numpy : 3.743171691894531e-05 nb_pixel_total : 160 time to create 1 rle with old method : 0.00030303001403808594 length of segment : 12 time for calcul the mask position with numpy : 9.918212890625e-05 nb_pixel_total : 280 time to create 1 rle with old method : 0.0009367465972900391 length of segment : 24 time for calcul the mask position with numpy : 5.5789947509765625e-05 nb_pixel_total : 392 time to create 1 rle with old method : 0.0006456375122070312 length of segment : 36 time for calcul the mask position with numpy : 7.796287536621094e-05 nb_pixel_total : 1525 time to create 1 rle with old method : 0.0019125938415527344 length of segment : 63 time for calcul the mask position with numpy : 4.124641418457031e-05 nb_pixel_total : 475 time to create 1 rle with old method : 0.0006191730499267578 length of segment : 29 time for calcul the mask position with numpy : 6.794929504394531e-05 nb_pixel_total : 1636 time to create 1 rle with old method : 0.002061128616333008 length of segment : 39 time for calcul the mask position with numpy : 3.981590270996094e-05 nb_pixel_total : 440 time to create 1 rle with old method : 0.0005786418914794922 length of segment : 27 time for calcul the mask position with numpy : 3.719329833984375e-05 nb_pixel_total : 278 time to create 1 rle with old method : 0.0003802776336669922 length of segment : 22 time for calcul the mask position with numpy : 3.5762786865234375e-05 nb_pixel_total : 274 time to create 1 rle with old method : 0.0003712177276611328 length of segment : 23 time for calcul the mask position with numpy : 4.0531158447265625e-05 nb_pixel_total : 441 time to create 1 rle with old method : 0.0006237030029296875 length of segment : 29 time for calcul the mask position with numpy : 4.553794860839844e-05 nb_pixel_total : 599 time to create 1 rle with old method : 0.0007958412170410156 length of segment : 43 time for calcul the mask position with numpy : 3.719329833984375e-05 nb_pixel_total : 94 time to create 1 rle with old method : 0.00019216537475585938 length of segment : 17 time for calcul the mask position with numpy : 5.364418029785156e-05 nb_pixel_total : 740 time to create 1 rle with old method : 0.0010218620300292969 length of segment : 28 time for calcul the mask position with numpy : 0.00010538101196289062 nb_pixel_total : 1216 time to create 1 rle with old method : 0.001623392105102539 length of segment : 62 time for calcul the mask position with numpy : 6.270408630371094e-05 nb_pixel_total : 356 time to create 1 rle with old method : 0.0006844997406005859 2025-02-20 17:20:53.217210: E tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-02-20 17:20:53.217415: F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:273] Unexpected Event status: 1 max_time_sub_proc : 3600 Catched exception ! Connect or reconnect ! in case -12 caffe_path_current : About to save ! 2 After save, about to update current ! 18.26user 13.05system 1:00:08elapsed 0%CPU (0avgtext+0avgdata 2295920maxresident)k 364344inputs+3872outputs (11717major+697840minor)pagefaults 0swaps