python /home/admin/mtr/script_for_cron.py -j python_test3 -m 12 -a ' --short_python3 -v ' -s python_test3 -M 0 -S 0 -U 100,100,120 import MySQLdb succeeded Import error (python version) python version = 3 warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data list_job_run_as_list : ['mask_detection', 'datou', 'CacheModelData_queries', 'CachePhotoData_queries', 'test_fork', 'prepare_maskdata', 'portfolio_queries', 'sla_mensuel'] python version used : 3 liste_fichiers : [('tests/mask_test', True, 'Test mask-detection ', 'mask_detection'), ('tests/datou_test', True, 'Datou All Test', 'datou', 'all'), ('mtr/database_queries/CacheModelData_queries', True, 'Test Cache Model Data', 'CacheModelData_queries'), ('tests/cache_photo_data_test', True, 'Test local_cache_photo ', 'CachePhotoData_queries'), ('mtr/mask_rcnn/prepare_maskdata', True, 'test prepare mask data', 'prepare_maskdata', 'all'), ('mtr/database_queries/portfolio_queries', True, 'test portfolio queries', 'portfolio_queries'), ('prod/memo/memo', True, 'SLA Mensuel', 'sla_mensuel', 'all')] #&_# BEGIN OF TEST : tests/mask_test #&_# /home/admin/workarea/git/Velours/python/tests/mask_test.py Test mask-detection python version used : 3 ############################### TEST memory used ################################ free memory at begining : begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 3738 run mask_detect Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.5484251976013184 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:mask_detect Wed Mar 5 00:35:28 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3738 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 /home/admin/workarea/git/Velours/python/tests/python_tests.py:11: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp 2025-03-05 00:35:31.572188: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-05 00:35:31.603133: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-05 00:35:31.605319: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f278c000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-05 00:35:31.605406: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-05 00:35:31.609920: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-05 00:35:31.878126: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x409cd400 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-05 00:35:31.878189: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-05 00:35:31.879292: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-05 00:35:31.879710: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 00:35:31.882889: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 00:35:31.885889: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-05 00:35:31.886391: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-05 00:35:31.889211: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-05 00:35:31.890584: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-05 00:35:31.895622: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-05 00:35:31.896803: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-05 00:35:31.896904: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 00:35:31.897513: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-05 00:35:31.897531: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-05 00:35:31.897542: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-05 00:35:31.898474: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3286 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl454 thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3473 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21)) {'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_coco_origin NUM_CLASSES 81 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 2025-03-05 00:35:43.101823: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-05 00:35:43.101916: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 00:35:43.101941: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 00:35:43.101963: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-05 00:35:43.101984: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-05 00:35:43.102005: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-05 00:35:43.102038: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-05 00:35:43.102060: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-05 00:35:43.103123: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-05 00:35:43.104104: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-05 00:35:43.104143: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 00:35:43.104165: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 00:35:43.104185: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-05 00:35:43.104204: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-05 00:35:43.104224: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-05 00:35:43.104244: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-05 00:35:43.104263: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-05 00:35:43.105304: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-05 00:35:43.105334: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-05 00:35:43.105345: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-05 00:35:43.105354: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-05 00:35:43.106446: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3286 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : mask_coco_origin model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-03-05 00:35:51.620807: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 00:35:51.808223: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-05 00:35:53.196182: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.196884: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.44G (2619644160 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.197452: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.20G (2357679616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.197477: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 00:35:53.198058: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.198074: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 00:35:53.205853: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.205877: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 00:35:53.206453: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.206469: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 00:35:53.213540: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.213562: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 466.56MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 00:35:53.214096: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.214111: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 466.56MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 00:35:53.246549: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.246583: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 00:35:53.247171: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.247190: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 00:35:53.253053: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.253074: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 243.25MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 00:35:53.253652: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.253668: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 243.25MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 00:35:53.288056: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.288667: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.290543: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.291149: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.331566: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.332170: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.334126: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.334712: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.341338: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.341932: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.346141: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.346740: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.357102: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.357689: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.359246: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.359829: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.365230: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.365786: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.367493: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.368076: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.373703: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.374315: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.375874: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.376421: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.402520: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.403123: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.403711: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.404267: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.407795: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.408351: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.423703: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.424282: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.424817: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.425352: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.437667: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.438216: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.438753: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.439344: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.443612: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.444223: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.448863: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.449457: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.461512: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.462108: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.466230: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.466775: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.467361: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.467944: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.489247: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.489818: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.490406: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.491047: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.491635: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.492211: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.492749: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.493289: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.545868: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.546462: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.577357: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.577406: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-03-05 00:35:53.578466: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.579572: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.589077: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.589812: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.590537: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.591290: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.610940: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.611875: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.639016: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.639636: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.640225: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.640799: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.647945: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.648526: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.649583: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.650599: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.651635: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.666107: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.666686: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.683954: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.684536: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.685122: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.685695: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.686276: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:35:53.686849: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.71G (2910715904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/mask_coco_origin /data/models_weight/mask_coco_origin/mask_model.h5 size_local : 257557808 size in s3 : 257557808 create time local : 2021-08-09 05:27:17 create time in s3 : 2021-08-06 19:45:17 mask_model.h5 already exist and didn't need to update list_images length : 1 NEW PHOTO Processing 1 images image shape: (480, 640, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 89) min: 0.00000 max: 640.00000 nb d'objets trouves : 5 Detection mask done ! Trying to reset tf kernel 2450100 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 1573 tf kernel not reseted sub process len(results) : 1 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 1 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 2766 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl454 Catched exception ! Connect or reconnect ! thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3473 ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] time for calcul the mask position with numpy : 0.0006284713745117188 nb_pixel_total : 11298 time to create 1 rle with old method : 0.013458251953125 length of segment : 248 time for calcul the mask position with numpy : 0.0032033920288085938 nb_pixel_total : 141839 time to create 1 rle with old method : 0.15759611129760742 length of segment : 364 time for calcul the mask position with numpy : 0.0003235340118408203 nb_pixel_total : 14487 time to create 1 rle with old method : 0.017154455184936523 length of segment : 152 time for calcul the mask position with numpy : 0.00020241737365722656 nb_pixel_total : 4978 time to create 1 rle with old method : 0.009626388549804688 length of segment : 51 time for calcul the mask position with numpy : 8.940696716308594e-05 nb_pixel_total : 1593 time to create 1 rle with old method : 0.002136707305908203 length of segment : 38 time spent for convertir_results : 1.0572237968444824 time spend for datou_step_exec : 29.229087352752686 time spend to save output : 7.62939453125e-05 total time spend for step 1 : 29.229163646697998 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : False eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 3314 chid ids of type : 445 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Number RLEs to save : 853 begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 time used for this insertion : 0.013991594314575195 save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {'957285035': [[(957285035, 492601069, 445, 19, 187, 11, 300, 0.9963844, [(146, 25, 4), (143, 26, 10), (141, 27, 12), (139, 28, 14), (137, 29, 17), (129, 30, 25), (124, 31, 30), (22, 32, 33), (118, 32, 36), (21, 33, 47), (114, 33, 40), (21, 34, 57), (111, 34, 44), (21, 35, 61), (83, 35, 6), (107, 35, 48), (20, 36, 73), (96, 36, 59), (20, 37, 135), (20, 38, 136), (20, 39, 136), (20, 40, 137), (20, 41, 137), (20, 42, 138), (19, 43, 139), (19, 44, 140), (19, 45, 140), (19, 46, 141), (19, 47, 141), (19, 48, 140), (19, 49, 139), (19, 50, 138), (19, 51, 136), (19, 52, 135), (19, 53, 133), (19, 54, 130), (19, 55, 128), (19, 56, 126), (19, 57, 124), (19, 58, 122), (19, 59, 117), (19, 60, 115), (19, 61, 113), (19, 62, 111), (19, 63, 108), (19, 64, 106), (19, 65, 104), (19, 66, 103), (19, 67, 102), (19, 68, 101), (19, 69, 99), (19, 70, 98), (19, 71, 97), (19, 72, 95), (19, 73, 94), (19, 74, 92), (19, 75, 91), (19, 76, 90), (19, 77, 89), (19, 78, 89), (19, 79, 88), (19, 80, 87), (19, 81, 86), (19, 82, 85), (19, 83, 84), (19, 84, 83), (19, 85, 83), (19, 86, 82), (19, 87, 81), (19, 88, 81), (19, 89, 80), (19, 90, 78), (19, 91, 77), (19, 92, 76), (19, 93, 75), (19, 94, 73), (19, 95, 72), (19, 96, 71), (19, 97, 70), (19, 98, 69), (19, 99, 69), (19, 100, 68), (19, 101, 67), (19, 102, 67), (19, 103, 66), (19, 104, 65), (19, 105, 65), (19, 106, 64), (19, 107, 63), (19, 108, 63), (19, 109, 62), (19, 110, 61), (19, 111, 60), (19, 112, 60), (19, 113, 59), (19, 114, 58), (19, 115, 57), (19, 116, 56), (19, 117, 55), (19, 118, 54), (19, 119, 53), (19, 120, 53), (19, 121, 52), (19, 122, 52), (19, 123, 51), (19, 124, 51), (19, 125, 50), (19, 126, 50), (19, 127, 49), (19, 128, 49), (19, 129, 49), (19, 130, 48), (19, 131, 48), (19, 132, 48), (19, 133, 47), (19, 134, 47), (19, 135, 46), (19, 136, 46), (19, 137, 45), (19, 138, 44), (19, 139, 44), (19, 140, 43), (19, 141, 42), (19, 142, 41), (19, 143, 40), (19, 144, 39), (19, 145, 39), (19, 146, 38), (19, 147, 37), (19, 148, 37), (19, 149, 37), (19, 150, 36), (19, 151, 36), (19, 152, 35), (19, 153, 34), (19, 154, 33), (19, 155, 32), (19, 156, 30), (19, 157, 29), (19, 158, 28), (19, 159, 27), (19, 160, 26), (19, 161, 25), (19, 162, 24), (19, 163, 24), (19, 164, 23), (19, 165, 23), (19, 166, 22), (19, 167, 21), (19, 168, 21), (19, 169, 20), (19, 170, 20), (19, 171, 19), (19, 172, 19), (19, 173, 19), (19, 174, 18), (19, 175, 18), (19, 176, 17), (19, 177, 17), (19, 178, 16), (19, 179, 16), (19, 180, 15), (19, 181, 15), (19, 182, 14), (19, 183, 14), (19, 184, 14), (19, 185, 13), (19, 186, 13), (19, 187, 13), (19, 188, 13), (19, 189, 13), (19, 190, 12), (19, 191, 12), (19, 192, 12), (20, 193, 11), (20, 194, 11), (20, 195, 10), (20, 196, 10), (20, 197, 10), (20, 198, 10), (20, 199, 9), (20, 200, 9), (20, 201, 9), (20, 202, 9), (20, 203, 8), (20, 204, 8), (20, 205, 8), (20, 206, 8), (20, 207, 8), (20, 208, 8), (20, 209, 7), (20, 210, 7), (20, 211, 7), (20, 212, 7), (20, 213, 7), (20, 214, 7), (20, 215, 7), (20, 216, 7), (20, 217, 7), (20, 218, 7), (20, 219, 8), (20, 220, 8), (20, 221, 8), (20, 222, 8), (20, 223, 8), (20, 224, 8), (20, 225, 7), (20, 226, 7), (20, 227, 7), (20, 228, 7), (20, 229, 7), (20, 230, 7), (20, 231, 7), (20, 232, 6), (20, 233, 6), (20, 234, 7), (20, 235, 7), (20, 236, 7), (20, 237, 7), (20, 238, 8), (20, 239, 8), (20, 240, 8), (20, 241, 8), (20, 242, 9), (20, 243, 9), (20, 244, 9), (20, 245, 9), (20, 246, 8), (20, 247, 8), (20, 248, 8), (20, 249, 8), (20, 250, 8), (20, 251, 8), (20, 252, 8), (20, 253, 8), (20, 254, 8), (21, 255, 7), (21, 256, 7), (21, 257, 7), (21, 258, 7), (21, 259, 8), (22, 260, 7), (22, 261, 7), (23, 262, 6), (24, 263, 5), (25, 264, 4), (26, 265, 2), (27, 266, 1)], ['27,266,22,261,20,254,20,193,19,192,19,43,20,36,22,32,54,32,55,33,67,33,68,34,77,34,78,35,88,35,95,37,96,36,106,36,118,32,123,32,129,30,136,30,146,25,152,26,154,37,159,47,151,53,149,53,140,58,136,58,129,62,123,64,119,68,110,74,101,84,99,88,92,93,81,107,78,112,71,119,67,127,64,136,55,147,54,151,42,162,31,185,29,195,26,209,26,225,27,241']), (957285035, 492601069, 445, 56, 577, 45, 423, 0.99409795, [(229, 46, 198), (204, 47, 248), (176, 48, 284), (171, 49, 294), (168, 50, 300), (165, 51, 306), (163, 52, 311), (161, 53, 315), (158, 54, 320), (153, 55, 326), (149, 56, 332), (144, 57, 339), (139, 58, 345), (137, 59, 349), (135, 60, 353), (133, 61, 356), (131, 62, 360), (129, 63, 364), (127, 64, 367), (126, 65, 369), (125, 66, 371), (124, 67, 373), (123, 68, 375), (121, 69, 378), (120, 70, 380), (118, 71, 383), (117, 72, 384), (115, 73, 387), (114, 74, 389), (112, 75, 392), (111, 76, 394), (109, 77, 396), (108, 78, 398), (106, 79, 401), (104, 80, 404), (102, 81, 407), (101, 82, 409), (100, 83, 411), (99, 84, 414), (98, 85, 416), (97, 86, 419), (96, 87, 421), (95, 88, 423), (94, 89, 425), (93, 90, 427), (93, 91, 427), (92, 92, 429), (91, 93, 431), (91, 94, 432), (90, 95, 434), (89, 96, 436), (88, 97, 438), (87, 98, 440), (86, 99, 442), (85, 100, 444), (84, 101, 447), (83, 102, 449), (82, 103, 451), (81, 104, 453), (80, 105, 455), (80, 106, 456), (79, 107, 457), (79, 108, 458), (78, 109, 460), (78, 110, 460), (78, 111, 461), (77, 112, 462), (77, 113, 463), (76, 114, 464), (76, 115, 465), (76, 116, 465), (75, 117, 467), (75, 118, 468), (75, 119, 468), (75, 120, 469), (75, 121, 470), (74, 122, 472), (74, 123, 473), (74, 124, 474), (74, 125, 476), (74, 126, 477), (74, 127, 478), (74, 128, 479), (73, 129, 481), (73, 130, 482), (73, 131, 483), (73, 132, 484), (73, 133, 484), (73, 134, 484), (72, 135, 486), (72, 136, 486), (72, 137, 487), (72, 138, 487), (72, 139, 488), (71, 140, 489), (71, 141, 490), (71, 142, 490), (71, 143, 491), (71, 144, 491), (70, 145, 493), (70, 146, 493), (70, 147, 493), (69, 148, 495), (68, 149, 496), (68, 150, 497), (67, 151, 498), (66, 152, 500), (65, 153, 501), (65, 154, 502), (64, 155, 504), (64, 156, 504), (63, 157, 506), (63, 158, 506), (62, 159, 508), (62, 160, 508), (62, 161, 508), (61, 162, 510), (61, 163, 510), (61, 164, 510), (61, 165, 510), (61, 166, 511), (61, 167, 511), (60, 168, 512), (60, 169, 512), (60, 170, 512), (60, 171, 513), (60, 172, 513), (60, 173, 513), (59, 174, 514), (59, 175, 514), (59, 176, 514), (59, 177, 513), (59, 178, 513), (58, 179, 514), (58, 180, 514), (58, 181, 514), (58, 182, 514), (58, 183, 514), (58, 184, 514), (57, 185, 514), (57, 186, 514), (57, 187, 514), (57, 188, 513), (57, 189, 513), (57, 190, 512), (57, 191, 512), (57, 192, 511), (57, 193, 511), (57, 194, 510), (57, 195, 509), (57, 196, 509), (57, 197, 508), (57, 198, 508), (57, 199, 507), (57, 200, 507), (57, 201, 506), (57, 202, 505), (57, 203, 505), (57, 204, 504), (57, 205, 504), (57, 206, 503), (57, 207, 502), (57, 208, 502), (57, 209, 501), (57, 210, 501), (57, 211, 500), (57, 212, 500), (57, 213, 499), (57, 214, 499), (57, 215, 498), (57, 216, 497), (57, 217, 496), (57, 218, 495), (57, 219, 494), (57, 220, 493), (57, 221, 491), (57, 222, 490), (57, 223, 489), (57, 224, 488), (57, 225, 487), (56, 226, 487), (56, 227, 486), (56, 228, 485), (56, 229, 484), (56, 230, 483), (56, 231, 482), (56, 232, 481), (56, 233, 479), (56, 234, 478), (56, 235, 476), (56, 236, 473), (56, 237, 471), (56, 238, 470), (56, 239, 468), (56, 240, 467), (56, 241, 466), (56, 242, 464), (56, 243, 463), (56, 244, 461), (56, 245, 459), (56, 246, 457), (56, 247, 454), (56, 248, 452), (56, 249, 451), (56, 250, 449), (56, 251, 448), (56, 252, 447), (56, 253, 446), (56, 254, 445), (56, 255, 444), (56, 256, 443), (56, 257, 441), (56, 258, 440), (56, 259, 438), (56, 260, 436), (56, 261, 434), (56, 262, 432), (56, 263, 431), (56, 264, 429), (56, 265, 428), (56, 266, 427), (56, 267, 426), (56, 268, 425), (56, 269, 424), (56, 270, 423), (56, 271, 421), (56, 272, 420), (56, 273, 418), (56, 274, 416), (56, 275, 414), (56, 276, 413), (56, 277, 411), (56, 278, 410), (56, 279, 409), (56, 280, 408), (56, 281, 407), (56, 282, 406), (56, 283, 405), (56, 284, 403), (56, 285, 402), (56, 286, 400), (56, 287, 398), (56, 288, 396), (56, 289, 394), (56, 290, 393), (56, 291, 391), (56, 292, 390), (57, 293, 388), (57, 294, 387), (57, 295, 386), (57, 296, 385), (57, 297, 383), (57, 298, 382), (57, 299, 380), (57, 300, 378), (57, 301, 376), (57, 302, 375), (57, 303, 373), (57, 304, 372), (57, 305, 370), (57, 306, 369), (57, 307, 368), (57, 308, 367), (58, 309, 365), (58, 310, 364), (58, 311, 363), (59, 312, 360), (59, 313, 359), (60, 314, 357), (60, 315, 356), (61, 316, 354), (62, 317, 352), (63, 318, 350), (63, 319, 350), (64, 320, 348), (65, 321, 347), (67, 322, 344), (69, 323, 342), (72, 324, 338), (74, 325, 336), (77, 326, 332), (80, 327, 329), (84, 328, 325), (87, 329, 321), (91, 330, 317), (94, 331, 313), (96, 332, 311), (98, 333, 309), (100, 334, 306), (102, 335, 304), (104, 336, 301), (106, 337, 299), (109, 338, 295), (111, 339, 293), (114, 340, 289), (116, 341, 287), (119, 342, 283), (122, 343, 280), (124, 344, 277), (126, 345, 274), (128, 346, 272), (130, 347, 269), (131, 348, 267), (133, 349, 265), (135, 350, 262), (137, 351, 260), (140, 352, 256), (142, 353, 253), (145, 354, 250), (147, 355, 248), (149, 356, 245), (151, 357, 243), (153, 358, 240), (155, 359, 238), (157, 360, 235), (159, 361, 233), (161, 362, 231), (163, 363, 229), (165, 364, 226), (167, 365, 224), (170, 366, 221), (172, 367, 219), (175, 368, 215), (178, 369, 212), (180, 370, 210), (181, 371, 209), (183, 372, 207), (184, 373, 205), (185, 374, 204), (186, 375, 203), (187, 376, 202), (189, 377, 199), (190, 378, 198), (192, 379, 195), (194, 380, 193), (196, 381, 190), (199, 382, 186), (202, 383, 183), (205, 384, 179), (207, 385, 176), (209, 386, 173), (211, 387, 170), (213, 388, 167), (215, 389, 165), (217, 390, 162), (220, 391, 158), (222, 392, 155), (225, 393, 152), (228, 394, 148), (231, 395, 144), (235, 396, 140), (240, 397, 134), (245, 398, 128), (249, 399, 124), (254, 400, 118), (259, 401, 113), (263, 402, 108), (267, 403, 103), (272, 404, 97), (280, 405, 87), (289, 406, 77), (298, 407, 65), (308, 408, 50), (317, 409, 34)], ['350,409,317,409,272,404,231,395,217,390,179,369,167,365,121,342,93,330,65,321,57,308,56,292,56,226,57,225,57,185,61,162,70,147,75,117,80,105,102,81,127,64,139,58,157,55,176,48,203,48,229,46,426,46,451,47,473,52,492,63,511,84,535,106,542,119,556,132,569,159,572,171,571,184,555,214,536,232,505,249,498,256,485,263,460,283,447,290,427,304,412,318,401,343,391,360,388,376,368,404']), (957285035, 492601069, 445, 501, 640, 10, 179, 0.9528609, [(544, 23, 17), (599, 23, 30), (533, 24, 101), (530, 25, 105), (527, 26, 108), (524, 27, 112), (521, 28, 115), (520, 29, 117), (518, 30, 120), (517, 31, 121), (515, 32, 123), (513, 33, 125), (511, 34, 127), (510, 35, 128), (509, 36, 129), (507, 37, 131), (506, 38, 132), (504, 39, 134), (503, 40, 135), (502, 41, 136), (502, 42, 136), (502, 43, 136), (502, 44, 136), (502, 45, 136), (501, 46, 137), (501, 47, 137), (501, 48, 137), (501, 49, 137), (501, 50, 137), (501, 51, 137), (501, 52, 137), (501, 53, 137), (501, 54, 137), (501, 55, 137), (501, 56, 137), (501, 57, 137), (501, 58, 137), (501, 59, 137), (501, 60, 137), (501, 61, 137), (501, 62, 137), (501, 63, 137), (501, 64, 137), (501, 65, 137), (501, 66, 137), (501, 67, 137), (502, 68, 136), (502, 69, 136), (502, 70, 136), (503, 71, 135), (503, 72, 135), (504, 73, 134), (505, 74, 133), (506, 75, 132), (508, 76, 131), (509, 77, 130), (510, 78, 129), (511, 79, 128), (512, 80, 127), (513, 81, 126), (514, 82, 125), (516, 83, 123), (517, 84, 122), (518, 85, 121), (518, 86, 121), (519, 87, 120), (520, 88, 119), (521, 89, 118), (521, 90, 118), (522, 91, 117), (523, 92, 116), (524, 93, 115), (525, 94, 114), (526, 95, 113), (527, 96, 112), (528, 97, 111), (529, 98, 110), (530, 99, 109), (531, 100, 108), (533, 101, 106), (534, 102, 105), (535, 103, 104), (536, 104, 103), (537, 105, 102), (540, 106, 99), (543, 107, 96), (545, 108, 94), (547, 109, 92), (548, 110, 91), (550, 111, 89), (551, 112, 88), (553, 113, 86), (554, 114, 85), (555, 115, 84), (556, 116, 83), (557, 117, 82), (558, 118, 81), (559, 119, 80), (560, 120, 79), (560, 121, 79), (561, 122, 78), (561, 123, 78), (561, 124, 78), (562, 125, 76), (562, 126, 76), (563, 127, 75), (563, 128, 75), (564, 129, 74), (564, 130, 74), (565, 131, 72), (565, 132, 72), (565, 133, 71), (565, 134, 71), (565, 135, 70), (566, 136, 69), (566, 137, 68), (566, 138, 67), (566, 139, 66), (566, 140, 65), (566, 141, 63), (566, 142, 61), (566, 143, 59), (566, 144, 58), (567, 145, 55), (567, 146, 54), (567, 147, 53), (567, 148, 51), (568, 149, 49), (568, 150, 49), (569, 151, 47), (569, 152, 46), (569, 153, 46), (569, 154, 45), (570, 155, 44), (570, 156, 43), (570, 157, 42), (571, 158, 41), (571, 159, 40), (572, 160, 38), (572, 161, 38), (572, 162, 37), (573, 163, 36), (573, 164, 35), (574, 165, 33), (575, 166, 32), (576, 167, 30), (578, 168, 27), (579, 169, 26), (582, 170, 22), (585, 171, 18), (588, 172, 14), (591, 173, 8)], ['598,173,591,173,587,171,579,169,573,164,572,160,569,154,569,151,566,144,566,136,565,131,561,124,560,120,552,112,542,106,537,105,532,100,531,100,521,90,518,85,503,72,501,67,501,46,502,41,511,34,516,32,521,28,526,27,533,24,543,24,544,23,560,23,561,24,598,24,599,23,628,23,633,24,637,30,637,75,638,76,638,124,637,130,634,136,630,140,625,142,616,149,613,155,609,160,608,163,601,172']), (957285035, 492601069, 445, 305, 479, 1, 57, 0.7547996, [(308, 2, 8), (330, 2, 15), (346, 2, 29), (307, 3, 121), (306, 4, 126), (306, 5, 128), (305, 6, 130), (305, 7, 132), (305, 8, 133), (305, 9, 135), (305, 10, 137), (305, 11, 138), (305, 12, 141), (305, 13, 143), (305, 14, 145), (305, 15, 147), (305, 16, 149), (305, 17, 150), (305, 18, 151), (305, 19, 153), (305, 20, 154), (305, 21, 155), (305, 22, 156), (305, 23, 157), (305, 24, 158), (305, 25, 159), (305, 26, 160), (305, 27, 161), (306, 28, 160), (306, 29, 161), (307, 30, 160), (308, 31, 6), (348, 31, 120), (368, 32, 101), (401, 33, 69), (411, 34, 60), (420, 35, 52), (425, 36, 47), (429, 37, 44), (432, 38, 41), (435, 39, 38), (437, 40, 37), (440, 41, 35), (442, 42, 34), (445, 43, 31), (448, 44, 28), (452, 45, 25), (455, 46, 22), (459, 47, 17), (463, 48, 12), (467, 49, 7)], ['473,49,467,49,466,48,463,48,462,47,455,46,451,44,447,43,442,42,439,40,437,40,434,38,432,38,428,36,420,35,419,34,411,34,410,33,401,33,400,32,368,32,367,31,348,31,347,30,314,30,313,31,308,31,305,27,305,6,308,2,315,2,316,3,329,3,330,2,374,2,375,3,427,3,431,4,438,9,441,10,443,12,445,12,453,16,457,19,465,27,466,30,471,35,472,39,475,42,476,46']), (957285035, 492601069, 445, 470, 548, 7, 48, 0.70224357, [(492, 9, 11), (520, 9, 7), (487, 10, 42), (487, 11, 44), (486, 12, 48), (475, 13, 8), (485, 13, 56), (471, 14, 71), (471, 15, 71), (471, 16, 72), (471, 17, 73), (471, 18, 75), (471, 19, 75), (471, 20, 71), (471, 21, 64), (472, 22, 58), (473, 23, 54), (471, 24, 54), (471, 25, 51), (471, 26, 49), (471, 27, 48), (471, 28, 46), (472, 29, 43), (472, 30, 42), (473, 31, 39), (473, 32, 38), (474, 33, 36), (475, 34, 34), (476, 35, 33), (477, 36, 30), (478, 37, 28), (479, 38, 25), (479, 39, 24), (480, 40, 21), (480, 41, 19), (481, 42, 16), (482, 43, 12), (482, 44, 5)], ['486,44,482,44,479,38,473,32,471,28,471,24,473,23,471,21,471,14,474,14,475,13,482,13,484,14,487,11,487,10,491,10,492,9,502,9,503,10,519,10,520,9,526,9,534,13,540,13,543,17,545,18,545,19,542,19,541,20,535,20,534,21,530,21,524,24,522,24,518,27,512,30,508,34,508,35,504,37,502,39,496,42,494,42,493,43,487,43'])], 'temp/1741131328_2449776_957285035_a42482e51c93c8025d243dd179aee85b.jpg']} free memory after detection : begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 2766 ############################### TEST detect object ################################ run mask_detect Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.3151869773864746 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:mask_detect Wed Mar 5 00:36:10 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 2766 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-03-05 00:36:13.099830: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-05 00:36:13.127106: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-05 00:36:13.128627: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f278c000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-05 00:36:13.128681: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-05 00:36:13.131460: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-05 00:36:13.414600: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x40edc5d0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-05 00:36:13.414651: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-05 00:36:13.415286: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-05 00:36:13.415685: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 00:36:13.421271: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 00:36:13.423677: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-05 00:36:13.424226: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-05 00:36:13.428835: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-05 00:36:13.430420: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-05 00:36:13.436039: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-05 00:36:13.437354: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-05 00:36:13.437461: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 00:36:13.438048: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-05 00:36:13.438064: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-05 00:36:13.438073: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-05 00:36:13.439061: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3507 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-03-05 00:36:13.524711: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-05 00:36:13.524814: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 00:36:13.524834: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 00:36:13.524851: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-05 00:36:13.524868: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-05 00:36:13.524885: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-05 00:36:13.524901: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-05 00:36:13.524930: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-05 00:36:13.525911: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-05 00:36:13.526923: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-05 00:36:13.526969: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 00:36:13.526986: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 00:36:13.527002: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-05 00:36:13.527018: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-05 00:36:13.527034: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-05 00:36:13.527049: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-05 00:36:13.527065: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-05 00:36:13.527952: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-05 00:36:13.527978: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-05 00:36:13.527986: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-05 00:36:13.527994: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-05 00:36:13.528933: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3507 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21)) {'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_coco_origin NUM_CLASSES 81 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : mask_coco_origin model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-03-05 00:36:20.869323: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 00:36:21.045910: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-05 00:36:22.451963: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.186432: W tensorflow/core/common_runtime/bfc_allocator.cc:311] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature. 2025-03-05 00:36:23.250782: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.250845: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.67GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 00:36:23.251979: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.252010: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.67GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 00:36:23.258964: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 00:36:23.258996: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 00:36:23.265573: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.265599: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.78GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 00:36:23.266442: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.266461: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.78GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 00:36:23.291063: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.291133: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 19.91MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 00:36:23.291152: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-03-05 00:36:23.292248: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.292274: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 16.00MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 00:36:23.293385: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.293412: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 16.00MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 00:36:23.300198: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.300216: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 63.85MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 00:36:23.300798: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.301393: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.302014: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.309823: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.310408: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.324667: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.325479: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.326299: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.327212: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.331343: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.331972: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.332594: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.333215: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.334210: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.344378: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.345003: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.355372: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.356002: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.356606: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.357190: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.357782: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:23.358366: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.93G (3142451200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/mask_coco_origin /data/models_weight/mask_coco_origin/mask_model.h5 size_local : 257557808 size in s3 : 257557808 create time local : 2021-08-09 05:27:17 create time in s3 : 2021-08-06 19:45:17 mask_model.h5 already exist and didn't need to update list_images length : 1 NEW PHOTO Processing 1 images image shape: (720, 1280, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 89) min: 0.00000 max: 1280.00000 nb d'objets trouves : 4 Detection mask done ! Trying to reset tf kernel 2452250 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 2766 tf kernel not reseted sub process len(results) : 1 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 1 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 3959 list_Values should be empty [] ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] time for calcul the mask position with numpy : 0.0005099773406982422 nb_pixel_total : 16902 time to create 1 rle with old method : 0.020626068115234375 length of segment : 107 time for calcul the mask position with numpy : 0.022487163543701172 nb_pixel_total : 480750 time to create 1 rle with new method : 0.03505849838256836 length of segment : 632 time for calcul the mask position with numpy : 0.0005958080291748047 nb_pixel_total : 36642 time to create 1 rle with old method : 0.04249858856201172 length of segment : 133 time for calcul the mask position with numpy : 0.0001068115234375 nb_pixel_total : 4793 time to create 1 rle with old method : 0.00602269172668457 length of segment : 51 time spent for convertir_results : 0.3098587989807129 time spend for datou_step_exec : 16.617504596710205 time spend to save output : 6.198883056640625e-05 total time spend for step 1 : 16.61756658554077 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : False eke 12-6-18 : saveMask need to be cleaned for new output ! Catched exception ! Connect or reconnect ! Number saved : None batch 1 Loaded 400 chid ids of type : 445 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Number RLEs to save : 0 begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 time used for this insertion : 0.012494802474975586 save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {'917855882': [[(917855882, 492601069, 445, 1092, 1280, 0, 108, 0.99883765, [(1205, 1, 58), (1165, 2, 105), (1159, 3, 113), (1149, 4, 124), (1113, 5, 161), (1100, 6, 174), (1097, 7, 177), (1095, 8, 179), (1095, 9, 179), (1095, 10, 179), (1095, 11, 179), (1095, 12, 179), (1095, 13, 179), (1095, 14, 178), (1095, 15, 178), (1095, 16, 178), (1095, 17, 178), (1095, 18, 177), (1095, 19, 177), (1095, 20, 177), (1095, 21, 177), (1095, 22, 177), (1095, 23, 178), (1095, 24, 178), (1095, 25, 178), (1095, 26, 179), (1095, 27, 179), (1095, 28, 180), (1095, 29, 181), (1095, 30, 182), (1095, 31, 183), (1095, 32, 183), (1095, 33, 184), (1095, 34, 184), (1096, 35, 183), (1096, 36, 183), (1096, 37, 184), (1097, 38, 183), (1097, 39, 183), (1097, 40, 183), (1098, 41, 182), (1098, 42, 182), (1098, 43, 182), (1099, 44, 181), (1099, 45, 181), (1099, 46, 181), (1100, 47, 180), (1100, 48, 180), (1101, 49, 179), (1101, 50, 179), (1102, 51, 178), (1102, 52, 178), (1103, 53, 177), (1103, 54, 177), (1104, 55, 176), (1104, 56, 176), (1104, 57, 176), (1104, 58, 176), (1105, 59, 175), (1105, 60, 175), (1105, 61, 175), (1105, 62, 175), (1105, 63, 175), (1106, 64, 174), (1106, 65, 174), (1106, 66, 174), (1106, 67, 174), (1106, 68, 174), (1106, 69, 174), (1106, 70, 174), (1106, 71, 174), (1106, 72, 174), (1106, 73, 174), (1107, 74, 173), (1107, 75, 173), (1107, 76, 173), (1107, 77, 173), (1107, 78, 173), (1107, 79, 173), (1108, 80, 172), (1108, 81, 172), (1109, 82, 171), (1110, 83, 170), (1110, 84, 170), (1111, 85, 169), (1112, 86, 168), (1113, 87, 166), (1114, 88, 165), (1115, 89, 164), (1117, 90, 162), (1120, 91, 159), (1138, 92, 141), (1146, 93, 133), (1154, 94, 125), (1167, 95, 112), (1177, 96, 102), (1183, 97, 95), (1185, 98, 93), (1187, 99, 90), (1188, 100, 55), (1264, 100, 12), (1190, 101, 50), (1191, 102, 46), (1194, 103, 40), (1197, 104, 34), (1202, 105, 25), (1207, 106, 16)], ['1222,106,1207,106,1206,105,1197,104,1191,102,1182,96,1176,95,1167,95,1166,94,1154,94,1153,93,1146,93,1145,92,1137,91,1120,91,1115,89,1110,84,1107,79,1106,73,1106,64,1104,55,1099,46,1095,34,1095,8,1100,6,1112,6,1113,5,1148,5,1149,4,1158,4,1165,2,1204,2,1205,1,1262,1,1269,2,1273,5,1273,13,1271,18,1271,22,1273,27,1277,31,1279,37,1279,86,1278,87,1278,96,1275,100,1264,100,1263,99,1243,99,1230,104']), (917855882, 492601069, 445, 52, 1128, 16, 668, 0.9977496, [(711, 22, 22), (926, 22, 46), (608, 23, 146), (894, 23, 103), (598, 24, 234), (850, 24, 158), (590, 25, 427), (582, 26, 444), (575, 27, 458), (569, 28, 466), (565, 29, 472), (560, 30, 480), (556, 31, 486), (550, 32, 495), (545, 33, 502), (538, 34, 512), (532, 35, 520), (527, 36, 527), (523, 37, 534), (518, 38, 541), (514, 39, 548), (510, 40, 554), (506, 41, 561), (503, 42, 566), (499, 43, 572), (496, 44, 577), (493, 45, 582), (491, 46, 585), (489, 47, 589), (487, 48, 592), (485, 49, 595), (483, 50, 598), (482, 51, 600), (481, 52, 602), (480, 53, 603), (479, 54, 605), (478, 55, 606), (476, 56, 608), (475, 57, 610), (474, 58, 611), (473, 59, 613), (472, 60, 614), (470, 61, 616), (469, 62, 618), (468, 63, 619), (466, 64, 621), (465, 65, 623), (464, 66, 624), (462, 67, 626), (461, 68, 628), (459, 69, 630), (458, 70, 631), (456, 71, 633), (455, 72, 635), (453, 73, 637), (452, 74, 638), (451, 75, 639), (449, 76, 641), (448, 77, 642), (447, 78, 643), (446, 79, 644), (445, 80, 645), (444, 81, 646), (442, 82, 648), (441, 83, 649), (440, 84, 650), (439, 85, 651), (438, 86, 652), (437, 87, 653), (436, 88, 654), (435, 89, 655), (434, 90, 656), (433, 91, 657), (432, 92, 658), (431, 93, 659), (430, 94, 660), (429, 95, 661), (428, 96, 662), (427, 97, 663), (425, 98, 665), (423, 99, 667), (421, 100, 669), (419, 101, 671), (417, 102, 673), (413, 103, 677), (410, 104, 680), (405, 105, 685), (401, 106, 689), (397, 107, 693), (392, 108, 698), (387, 109, 703), (382, 110, 708), (377, 111, 713), (373, 112, 717), (369, 113, 721), (365, 114, 725), (362, 115, 728), (358, 116, 732), (356, 117, 734), (353, 118, 737), (351, 119, 739), (349, 120, 741), (346, 121, 744), (344, 122, 746), (341, 123, 749), (338, 124, 752), (335, 125, 755), (331, 126, 759), (327, 127, 763), (323, 128, 767), (319, 129, 770), (314, 130, 775), (308, 131, 781), (303, 132, 786), (294, 133, 795), (286, 134, 803), (279, 135, 810), (273, 136, 816), (267, 137, 822), (262, 138, 827), (258, 139, 831), (255, 140, 834), (252, 141, 837), (250, 142, 839), (247, 143, 842), (245, 144, 844), (242, 145, 847), (240, 146, 849), (237, 147, 852), (233, 148, 856), (230, 149, 859), (226, 150, 863), (220, 151, 869), (213, 152, 876), (207, 153, 882), (200, 154, 889), (193, 155, 896), (187, 156, 902), (183, 157, 906), (181, 158, 908), (178, 159, 911), (176, 160, 913), (174, 161, 915), (172, 162, 917), (170, 163, 919), (168, 164, 921), (167, 165, 922), (165, 166, 924), (164, 167, 925), (162, 168, 927), (161, 169, 928), (159, 170, 930), (157, 171, 932), (155, 172, 934), (153, 173, 935), (151, 174, 937), (148, 175, 940), (146, 176, 942), (144, 177, 944), (142, 178, 946), (140, 179, 948), (139, 180, 949), (137, 181, 951), (136, 182, 952), (134, 183, 954), (133, 184, 955), (132, 185, 956), (131, 186, 957), (130, 187, 958), (129, 188, 959), (128, 189, 960), (127, 190, 960), (126, 191, 961), (126, 192, 961), (125, 193, 962), (124, 194, 963), (123, 195, 964), (122, 196, 965), (122, 197, 965), (121, 198, 966), (120, 199, 967), (119, 200, 968), (118, 201, 969), (117, 202, 970), (116, 203, 971), (114, 204, 973), (113, 205, 973), (112, 206, 974), (111, 207, 975), (109, 208, 977), (108, 209, 978), (107, 210, 979), (106, 211, 980), (105, 212, 981), (104, 213, 982), (103, 214, 983), (102, 215, 984), (101, 216, 985), (101, 217, 984), (100, 218, 985), (100, 219, 985), (99, 220, 986), (98, 221, 987), (98, 222, 987), (97, 223, 988), (97, 224, 987), (96, 225, 988), (96, 226, 988), (95, 227, 989), (95, 228, 989), (94, 229, 990), (94, 230, 990), (94, 231, 990), (93, 232, 990), (93, 233, 990), (92, 234, 991), (92, 235, 991), (92, 236, 991), (91, 237, 992), (91, 238, 991), (91, 239, 991), (91, 240, 991), (91, 241, 990), (90, 242, 991), (90, 243, 990), (90, 244, 990), (90, 245, 989), (90, 246, 989), (89, 247, 990), (89, 248, 989), (89, 249, 989), (89, 250, 988), (89, 251, 988), (88, 252, 988), (88, 253, 988), (88, 254, 987), (88, 255, 986), (88, 256, 986), (87, 257, 986), (87, 258, 986), (87, 259, 985), (87, 260, 984), (87, 261, 983), (86, 262, 983), (86, 263, 983), (86, 264, 982), (86, 265, 981), (85, 266, 981), (85, 267, 980), (85, 268, 980), (84, 269, 980), (84, 270, 979), (84, 271, 979), (84, 272, 978), (83, 273, 979), (83, 274, 978), (83, 275, 977), (82, 276, 978), (82, 277, 977), (82, 278, 977), (81, 279, 977), (81, 280, 977), (81, 281, 977), (80, 282, 977), (80, 283, 977), (80, 284, 976), (79, 285, 977), (79, 286, 976), (79, 287, 976), (78, 288, 976), (78, 289, 976), (78, 290, 975), (77, 291, 976), (77, 292, 975), (77, 293, 975), (76, 294, 975), (76, 295, 975), (76, 296, 974), (75, 297, 975), (75, 298, 974), (74, 299, 975), (74, 300, 974), (74, 301, 974), (73, 302, 974), (73, 303, 974), (72, 304, 974), (72, 305, 974), (71, 306, 974), (71, 307, 973), (71, 308, 972), (70, 309, 972), (70, 310, 971), (70, 311, 970), (70, 312, 968), (69, 313, 968), (69, 314, 966), (69, 315, 964), (69, 316, 962), (68, 317, 961), (68, 318, 959), (68, 319, 958), (68, 320, 956), (67, 321, 955), (67, 322, 954), (67, 323, 952), (67, 324, 951), (66, 325, 951), (66, 326, 950), (66, 327, 948), (66, 328, 947), (65, 329, 947), (65, 330, 946), (65, 331, 946), (65, 332, 945), (65, 333, 944), (65, 334, 942), (65, 335, 941), (65, 336, 940), (65, 337, 939), (65, 338, 938), (64, 339, 937), (64, 340, 936), (64, 341, 934), (64, 342, 932), (64, 343, 930), (64, 344, 928), (64, 345, 926), (64, 346, 925), (64, 347, 923), (64, 348, 922), (64, 349, 920), (64, 350, 919), (63, 351, 919), (63, 352, 918), (63, 353, 917), (63, 354, 916), (63, 355, 915), (63, 356, 914), (63, 357, 912), (63, 358, 911), (63, 359, 910), (63, 360, 909), (63, 361, 908), (63, 362, 906), (63, 363, 905), (63, 364, 904), (63, 365, 902), (63, 366, 901), (63, 367, 899), (63, 368, 898), (63, 369, 896), (62, 370, 895), (62, 371, 893), (62, 372, 891), (62, 373, 890), (62, 374, 888), (62, 375, 887), (62, 376, 886), (62, 377, 885), (62, 378, 884), (62, 379, 883), (63, 380, 880), (63, 381, 879), (63, 382, 878), (63, 383, 877), (63, 384, 876), (63, 385, 875), (63, 386, 874), (63, 387, 873), (63, 388, 872), (64, 389, 870), (64, 390, 869), (64, 391, 868), (64, 392, 867), (64, 393, 865), (64, 394, 864), (64, 395, 863), (65, 396, 861), (65, 397, 860), (65, 398, 859), (65, 399, 858), (65, 400, 857), (65, 401, 856), (65, 402, 854), (65, 403, 853), (65, 404, 851), (65, 405, 850), (65, 406, 848), (66, 407, 846), (66, 408, 844), (66, 409, 843), (66, 410, 842), (66, 411, 841), (66, 412, 840), (66, 413, 838), (66, 414, 837), (66, 415, 836), (66, 416, 835), (66, 417, 835), (66, 418, 834), (66, 419, 833), (67, 420, 831), (67, 421, 830), (67, 422, 829), (67, 423, 829), (67, 424, 828), (67, 425, 827), (67, 426, 826), (67, 427, 825), (67, 428, 824), (68, 429, 822), (68, 430, 820), (68, 431, 819), (68, 432, 818), (68, 433, 816), (68, 434, 815), (68, 435, 813), (68, 436, 811), (69, 437, 809), (69, 438, 807), (69, 439, 806), (69, 440, 804), (69, 441, 803), (69, 442, 802), (69, 443, 800), (70, 444, 798), (70, 445, 797), (70, 446, 796), (70, 447, 796), (71, 448, 794), (71, 449, 794), (72, 450, 792), (72, 451, 792), (73, 452, 790), (73, 453, 789), (74, 454, 788), (74, 455, 787), (75, 456, 786), (75, 457, 785), (76, 458, 784), (76, 459, 783), (77, 460, 782), (77, 461, 781), (77, 462, 781), (78, 463, 779), (78, 464, 779), (79, 465, 777), (79, 466, 777), (79, 467, 776), (80, 468, 775), (80, 469, 774), (80, 470, 774), (81, 471, 772), (81, 472, 771), (82, 473, 770), (82, 474, 769), (83, 475, 767), (83, 476, 766), (83, 477, 766), (84, 478, 764), (84, 479, 763), (85, 480, 761), (85, 481, 760), (85, 482, 759), (86, 483, 757), (86, 484, 755), (87, 485, 753), (87, 486, 752), (87, 487, 751), (88, 488, 748), (88, 489, 747), (88, 490, 746), (89, 491, 744), (89, 492, 743), (90, 493, 741), (90, 494, 741), (91, 495, 739), (91, 496, 738), (92, 497, 737), (93, 498, 735), (94, 499, 733), (94, 500, 733), (95, 501, 731), (96, 502, 729), (97, 503, 728), (98, 504, 726), (99, 505, 724), (99, 506, 724), (100, 507, 722), (101, 508, 721), (102, 509, 719), (104, 510, 717), (105, 511, 715), (106, 512, 714), (107, 513, 712), (108, 514, 711), (110, 515, 708), (111, 516, 707), (113, 517, 704), (114, 518, 703), (115, 519, 701), (117, 520, 698), (118, 521, 697), (119, 522, 695), (121, 523, 693), (122, 524, 691), (124, 525, 689), (125, 526, 687), (126, 527, 685), (128, 528, 683), (129, 529, 681), (131, 530, 678), (132, 531, 676), (134, 532, 673), (135, 533, 672), (137, 534, 669), (138, 535, 667), (140, 536, 664), (141, 537, 662), (143, 538, 659), (144, 539, 657), (146, 540, 654), (148, 541, 651), (149, 542, 649), (151, 543, 645), (153, 544, 642), (154, 545, 640), (156, 546, 638), (158, 547, 635), (159, 548, 633), (161, 549, 630), (162, 550, 628), (164, 551, 626), (166, 552, 623), (167, 553, 621), (169, 554, 618), (170, 555, 617), (171, 556, 615), (173, 557, 613), (174, 558, 611), (176, 559, 608), (177, 560, 607), (178, 561, 605), (180, 562, 603), (181, 563, 601), (183, 564, 599), (185, 565, 597), (186, 566, 595), (189, 567, 592), (192, 568, 589), (195, 569, 585), (198, 570, 582), (201, 571, 579), (204, 572, 575), (206, 573, 573), (209, 574, 569), (212, 575, 566), (215, 576, 563), (218, 577, 559), (221, 578, 556), (223, 579, 553), (226, 580, 550), (228, 581, 547), (230, 582, 545), (232, 583, 542), (234, 584, 540), (235, 585, 539), (237, 586, 536), (238, 587, 534), (240, 588, 531), (242, 589, 528), (243, 590, 526), (245, 591, 523), (247, 592, 520), (249, 593, 516), (251, 594, 513), (253, 595, 510), (256, 596, 505), (258, 597, 501), (261, 598, 497), (263, 599, 493), (267, 600, 488), (271, 601, 482), (274, 602, 478), (278, 603, 473), (281, 604, 468), (284, 605, 464), (287, 606, 460), (290, 607, 456), (292, 608, 453), (295, 609, 449), (297, 610, 446), (300, 611, 442), (303, 612, 438), (305, 613, 434), (307, 614, 431), (310, 615, 427), (312, 616, 423), (315, 617, 418), (317, 618, 415), (320, 619, 410), (322, 620, 406), (325, 621, 401), (327, 622, 396), (330, 623, 390), (333, 624, 384), (335, 625, 379), (338, 626, 374), (341, 627, 369), (345, 628, 362), (349, 629, 356), (353, 630, 350), (357, 631, 344), (360, 632, 340), (364, 633, 334), (368, 634, 328), (373, 635, 320), (378, 636, 313), (383, 637, 305), (389, 638, 295), (395, 639, 282), (401, 640, 270), (408, 641, 256), (416, 642, 240), (431, 643, 217), (448, 644, 193), (465, 645, 169), (480, 646, 148), (495, 647, 126), (511, 648, 104), (526, 649, 82), (565, 650, 9)], ['526,649,416,642,341,627,289,606,263,599,220,577,186,566,144,539,102,509,91,496,70,447,62,379,65,329,86,265,91,237,101,216,134,183,187,156,225,151,262,138,302,133,343,123,358,116,416,103,493,45,527,36,608,23,754,24,893,24,926,22,996,23,1032,27,1082,52,1089,72,1088,172,1082,237,1045,305,1019,322,1002,338,950,373,895,423,865,446,851,473,822,505,810,528,786,554,773,585,740,612,683,638,607,649']), (917855882, 492601069, 445, 0, 440, 0, 116, 0.9919425, [(127, 1, 141), (94, 2, 206), (384, 2, 2), (59, 3, 273), (340, 3, 57), (22, 4, 381), (19, 5, 387), (16, 6, 392), (15, 7, 394), (14, 8, 396), (14, 9, 397), (13, 10, 399), (12, 11, 400), (12, 12, 400), (11, 13, 402), (10, 14, 403), (11, 15, 403), (11, 16, 404), (12, 17, 403), (12, 18, 404), (12, 19, 405), (12, 20, 405), (12, 21, 406), (12, 22, 406), (12, 23, 407), (12, 24, 407), (12, 25, 408), (12, 26, 408), (12, 27, 408), (12, 28, 408), (12, 29, 409), (12, 30, 409), (12, 31, 409), (12, 32, 409), (12, 33, 409), (12, 34, 410), (12, 35, 410), (12, 36, 410), (12, 37, 410), (12, 38, 410), (12, 39, 410), (12, 40, 410), (12, 41, 411), (12, 42, 411), (12, 43, 411), (12, 44, 411), (12, 45, 411), (12, 46, 410), (12, 47, 410), (12, 48, 410), (12, 49, 410), (12, 50, 410), (12, 51, 410), (12, 52, 409), (12, 53, 408), (12, 54, 408), (12, 55, 407), (12, 56, 406), (12, 57, 404), (12, 58, 403), (11, 59, 403), (11, 60, 402), (11, 61, 401), (11, 62, 400), (11, 63, 400), (11, 64, 399), (11, 65, 398), (11, 66, 397), (11, 67, 397), (11, 68, 396), (11, 69, 395), (11, 70, 395), (11, 71, 394), (11, 72, 394), (11, 73, 394), (11, 74, 393), (11, 75, 393), (11, 76, 393), (11, 77, 393), (11, 78, 393), (11, 79, 393), (11, 80, 392), (10, 81, 394), (10, 82, 394), (10, 83, 395), (9, 84, 396), (9, 85, 262), (279, 85, 126), (9, 86, 75), (98, 86, 28), (142, 86, 117), (292, 86, 112), (9, 87, 71), (152, 87, 103), (294, 87, 110), (8, 88, 68), (161, 88, 91), (296, 88, 107), (8, 89, 63), (176, 89, 73), (297, 89, 106), (7, 90, 61), (205, 90, 40), (298, 90, 104), (7, 91, 57), (299, 91, 103), (6, 92, 54), (300, 92, 102), (6, 93, 50), (301, 93, 100), (7, 94, 46), (303, 94, 97), (7, 95, 44), (306, 95, 92), (7, 96, 42), (308, 96, 89), (7, 97, 40), (310, 97, 86), (7, 98, 38), (312, 98, 83), (8, 99, 34), (314, 99, 79), (8, 100, 32), (317, 100, 75), (8, 101, 29), (319, 101, 71), (13, 102, 19), (324, 102, 63), (20, 103, 6), (330, 103, 51), (337, 104, 37), (344, 105, 22), (352, 106, 3)], ['344,105,319,101,301,93,291,85,259,85,244,90,205,90,204,89,176,89,161,88,141,85,126,85,125,86,98,86,84,85,56,92,36,101,26,102,8,101,6,92,11,80,11,59,12,58,12,17,10,14,16,6,22,4,58,4,59,3,93,3,94,2,126,2,127,1,267,1,268,2,331,3,396,3,407,6,411,10,419,25,421,34,421,51,410,62,404,71,402,80,404,85,401,92,394,98,386,102,365,105']), (917855882, 492601069, 445, 390, 550, 0, 54, 0.9392051, [(414, 0, 7), (441, 0, 60), (508, 0, 28), (402, 1, 142), (401, 2, 146), (402, 3, 145), (404, 4, 143), (406, 5, 140), (408, 6, 137), (410, 7, 134), (411, 8, 132), (412, 9, 130), (413, 10, 127), (414, 11, 125), (415, 12, 123), (415, 13, 122), (416, 14, 120), (417, 15, 117), (417, 16, 116), (418, 17, 114), (418, 18, 113), (418, 19, 111), (418, 20, 109), (419, 21, 107), (419, 22, 105), (419, 23, 103), (419, 24, 102), (420, 25, 99), (420, 26, 97), (420, 27, 95), (420, 28, 94), (421, 29, 91), (421, 30, 90), (422, 31, 88), (422, 32, 88), (422, 33, 87), (423, 34, 84), (423, 35, 82), (423, 36, 81), (424, 37, 79), (424, 38, 77), (424, 39, 75), (424, 40, 73), (424, 41, 71), (425, 42, 67), (425, 43, 66), (426, 44, 62), (426, 45, 6), (433, 45, 52), (443, 46, 30), (450, 47, 1)], ['449,46,443,46,442,45,426,45,424,41,424,37,423,36,422,31,420,28,420,25,419,24,419,21,418,20,418,17,417,15,409,6,402,3,402,1,413,1,414,0,420,0,421,1,440,1,441,0,500,0,501,1,507,1,508,0,535,0,536,1,543,1,546,2,546,4,542,8,530,18,527,19,525,21,522,22,520,24,512,28,508,33,505,34,502,37,494,41,492,41,490,43,488,43,484,45,473,45,472,46'])], 'temp/1741131370_2449776_917855882_da0fa7b7e6b5b551fe26c0ba8713276d.jpg']} ############################### TEST POLYGON ################################ Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.23048615455627441 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:mask_detect Wed Mar 5 00:36:34 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 4180 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-03-05 00:36:37.556008: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-05 00:36:37.583136: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-05 00:36:37.585214: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f278c000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-05 00:36:37.585245: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-05 00:36:37.589295: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-05 00:36:37.853062: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x41cd9280 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-05 00:36:37.853119: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-05 00:36:37.854320: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-05 00:36:37.854753: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 00:36:37.857980: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 00:36:37.860868: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-05 00:36:37.861279: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-05 00:36:37.864071: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-05 00:36:37.865383: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-05 00:36:37.871791: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-05 00:36:37.872936: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-05 00:36:37.873021: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 00:36:37.873579: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-05 00:36:37.873594: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-05 00:36:37.873603: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-05 00:36:37.874551: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3728 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-03-05 00:36:37.962215: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-05 00:36:37.962307: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 00:36:37.962342: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 00:36:37.962364: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-05 00:36:37.962385: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-05 00:36:37.962406: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-05 00:36:37.962426: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-05 00:36:37.962447: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-05 00:36:37.963554: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-05 00:36:37.964580: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-05 00:36:37.964619: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 00:36:37.964640: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 00:36:37.964660: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-05 00:36:37.964679: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-05 00:36:37.964699: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-05 00:36:37.964719: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-05 00:36:37.964738: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-05 00:36:37.965822: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-05 00:36:37.965852: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-05 00:36:37.965863: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-05 00:36:37.965872: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-05 00:36:37.967015: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3728 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21)) {'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_coco_origin NUM_CLASSES 81 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : mask_coco_origin model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-03-05 00:36:46.654008: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 00:36:46.824799: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-05 00:36:48.240227: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.14G (3374186496 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 00:36:49.042163: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 00:36:49.042214: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. local folder : /data/models_weight/mask_coco_origin /data/models_weight/mask_coco_origin/mask_model.h5 size_local : 257557808 size in s3 : 257557808 create time local : 2021-08-09 05:27:17 create time in s3 : 2021-08-06 19:45:17 mask_model.h5 already exist and didn't need to update list_images length : 1 NEW PHOTO Processing 1 images image shape: (2448, 2448, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 89) min: 0.00000 max: 2448.00000 nb d'objets trouves : 1 Detection mask done ! Trying to reset tf kernel 2454018 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 89 tf kernel not reseted sub process len(results) : 1 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 1 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 4180 list_Values should be empty [] ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] time for calcul the mask position with numpy : 0.2628366947174072 nb_pixel_total : 3689278 time to create 1 rle with new method : 0.46739840507507324 length of segment : 2038 time spent for convertir_results : 2.1080548763275146 time spend for datou_step_exec : 19.861791849136353 time spend to save output : 3.266334533691406e-05 total time spend for step 1 : 19.86182451248169 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : False eke 12-6-18 : saveMask need to be cleaned for new output ! Catched exception ! Connect or reconnect ! Number saved : None batch 1 Loaded 721 chid ids of type : 445 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Number RLEs to save : 0 begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 time used for this insertion : 0.012514591217041016 save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {'917877156': [[(917877156, 492601069, 445, 7, 2268, 122, 2241, 0.9850345, [(523, 124, 444), (1148, 124, 279), (502, 125, 949), (481, 126, 993), (461, 127, 1035), (443, 128, 1091), (427, 129, 1149), (412, 130, 1177), (397, 131, 1201), (382, 132, 1224), (370, 133, 1243), (367, 134, 1254), (364, 135, 1264), (362, 136, 1273), (359, 137, 1282), (357, 138, 1291), (354, 139, 1300), (352, 140, 1308), (350, 141, 1315), (348, 142, 1321), (345, 143, 1328), (343, 144, 1334), (342, 145, 1339), (340, 146, 1344), (338, 147, 1350), (336, 148, 1355), (334, 149, 1361), (333, 150, 1365), (331, 151, 1370), (329, 152, 1375), (328, 153, 1379), (327, 154, 1383), (325, 155, 1388), (324, 156, 1392), (322, 157, 1396), (321, 158, 1400), (320, 159, 1404), (319, 160, 1407), (317, 161, 1412), (316, 162, 1415), (315, 163, 1419), (314, 164, 1422), (312, 165, 1427), (311, 166, 1431), (309, 167, 1436), (308, 168, 1439), (307, 169, 1442), (305, 170, 1446), (304, 171, 1449), (302, 172, 1454), (300, 173, 1458), (299, 174, 1462), (297, 175, 1466), (295, 176, 1471), (293, 177, 1476), (291, 178, 1481), (289, 179, 1486), (287, 180, 1491), (284, 181, 1497), (282, 182, 1502), (279, 183, 1508), (276, 184, 1515), (274, 185, 1520), (271, 186, 1527), (268, 187, 1534), (266, 188, 1540), (263, 189, 1547), (260, 190, 1554), (257, 191, 1561), (255, 192, 1568), (252, 193, 1577), (249, 194, 1586), (246, 195, 1596), (243, 196, 1605), (240, 197, 1614), (238, 198, 1622), (235, 199, 1631), (232, 200, 1640), (229, 201, 1648), (226, 202, 1657), (223, 203, 1666), (220, 204, 1675), (217, 205, 1683), (214, 206, 1691), (211, 207, 1696), (208, 208, 1701), (206, 209, 1705), (205, 210, 1707), (203, 211, 1711), (201, 212, 1715), (199, 213, 1718), (198, 214, 1721), (196, 215, 1724), (195, 216, 1727), (193, 217, 1730), (192, 218, 1732), (190, 219, 1735), (189, 220, 1738), (188, 221, 1740), (186, 222, 1743), (185, 223, 1745), (184, 224, 1747), (183, 225, 1749), (182, 226, 1751), (181, 227, 1753), (180, 228, 1755), (179, 229, 1757), (178, 230, 1759), (177, 231, 1761), (176, 232, 1763), (175, 233, 1765), (174, 234, 1767), (173, 235, 1768), (172, 236, 1770), (171, 237, 1772), (170, 238, 1774), (169, 239, 1775), (168, 240, 1777), (167, 241, 1779), (167, 242, 1780), (166, 243, 1781), (165, 244, 1783), (164, 245, 1785), (163, 246, 1787), (162, 247, 1789), (160, 248, 1792), (159, 249, 1794), (158, 250, 1796), (157, 251, 1797), (156, 252, 1799), (155, 253, 1802), (154, 254, 1804), (152, 255, 1807), (151, 256, 1809), (150, 257, 1811), (148, 258, 1814), (147, 259, 1816), (146, 260, 1818), (144, 261, 1822), (143, 262, 1824), (141, 263, 1827), (140, 264, 1830), (138, 265, 1833), (137, 266, 1835), (135, 267, 1839), (133, 268, 1842), (131, 269, 1846), (130, 270, 1849), (128, 271, 1852), (126, 272, 1856), (125, 273, 1859), (124, 274, 1862), (122, 275, 1865), (121, 276, 1868), (120, 277, 1871), (119, 278, 1873), (117, 279, 1877), (116, 280, 1879), (115, 281, 1881), (114, 282, 1884), (113, 283, 1886), (112, 284, 1888), (111, 285, 1890), (110, 286, 1893), (109, 287, 1895), (108, 288, 1897), (107, 289, 1899), (107, 290, 1900), (106, 291, 1902), (105, 292, 1904), (104, 293, 1906), (103, 294, 1908), (103, 295, 1909), (102, 296, 1911), (101, 297, 1913), (100, 298, 1915), (100, 299, 1915), (99, 300, 1917), (98, 301, 1919), (98, 302, 1920), (97, 303, 1922), (97, 304, 1922), (96, 305, 1924), (95, 306, 1926), (95, 307, 1926), (94, 308, 1928), (94, 309, 1929), (93, 310, 1930), (93, 311, 1931), (92, 312, 1933), (92, 313, 1933), (92, 314, 1934), (92, 315, 1934), (91, 316, 1936), (91, 317, 1936), (91, 318, 1937), (91, 319, 1937), (90, 320, 1939), (90, 321, 1939), (90, 322, 1940), (89, 323, 1941), (89, 324, 1942), (89, 325, 1942), (89, 326, 1943), (88, 327, 1945), (88, 328, 1945), (88, 329, 1946), (88, 330, 1946), (87, 331, 1948), (87, 332, 1949), (87, 333, 1949), (87, 334, 1950), (86, 335, 1951), (86, 336, 1952), (86, 337, 1953), (85, 338, 1955), (85, 339, 1955), (85, 340, 1956), (85, 341, 1957), (84, 342, 1958), (84, 343, 1959), (84, 344, 1960), (83, 345, 1962), (83, 346, 1962), (83, 347, 1963), (83, 348, 1964), (82, 349, 1966), (82, 350, 1967), (82, 351, 1967), (81, 352, 1969), (81, 353, 1970), (81, 354, 1971), (81, 355, 1972), (80, 356, 1974), (80, 357, 1975), (80, 358, 1976), (79, 359, 1978), (79, 360, 1979), (79, 361, 1980), (78, 362, 1982), (78, 363, 1983), (78, 364, 1984), (78, 365, 1985), (77, 366, 1987), (77, 367, 1988), (77, 368, 1989), (76, 369, 1992), (76, 370, 1993), (76, 371, 1994), (75, 372, 1996), (75, 373, 1997), (75, 374, 1998), (74, 375, 1999), (74, 376, 2000), (74, 377, 2001), (73, 378, 2003), (73, 379, 2004), (73, 380, 2005), (72, 381, 2007), (72, 382, 2007), (72, 383, 2008), (72, 384, 2009), (71, 385, 2011), (71, 386, 2011), (71, 387, 2012), (70, 388, 2014), (70, 389, 2014), (70, 390, 2015), (69, 391, 2016), (69, 392, 2017), (69, 393, 2017), (68, 394, 2019), (68, 395, 2019), (68, 396, 2020), (67, 397, 2021), (67, 398, 2022), (67, 399, 2022), (66, 400, 2024), (66, 401, 2024), (66, 402, 2025), (65, 403, 2026), (65, 404, 2026), (65, 405, 2027), (64, 406, 2028), (64, 407, 2029), (63, 408, 2030), (63, 409, 2031), (63, 410, 2031), (62, 411, 2033), (62, 412, 2033), (62, 413, 2034), (61, 414, 2035), (61, 415, 2035), (60, 416, 2037), (60, 417, 2037), (60, 418, 2038), (59, 419, 2039), (59, 420, 2040), (58, 421, 2041), (58, 422, 2041), (57, 423, 2043), (57, 424, 2043), (57, 425, 2044), (56, 426, 2045), (56, 427, 2045), (55, 428, 2047), (55, 429, 2047), (54, 430, 2049), (54, 431, 2049), (53, 432, 2050), (53, 433, 2051), (52, 434, 2052), (52, 435, 2053), (51, 436, 2054), (51, 437, 2054), (50, 438, 2056), (50, 439, 2056), (49, 440, 2057), (49, 441, 2058), (48, 442, 2059), (48, 443, 2059), (47, 444, 2061), (46, 445, 2062), (46, 446, 2063), (45, 447, 2064), (45, 448, 2064), (44, 449, 2066), (44, 450, 2066), (43, 451, 2067), (43, 452, 2068), (42, 453, 2069), (42, 454, 2069), (42, 455, 2070), (41, 456, 2071), (41, 457, 2071), (40, 458, 2073), (40, 459, 2073), (39, 460, 2074), (39, 461, 2075), (38, 462, 2076), (38, 463, 2076), (38, 464, 2077), (38, 465, 2077), (38, 466, 2077), (38, 467, 2077), (37, 468, 2079), (37, 469, 2079), (37, 470, 2079), (37, 471, 2080), (37, 472, 2080), (37, 473, 2080), (36, 474, 2082), (36, 475, 2082), (36, 476, 2082), (36, 477, 2083), (36, 478, 2083), (36, 479, 2083), (36, 480, 2084), (35, 481, 2085), (35, 482, 2086), (35, 483, 2086), (35, 484, 2086), (35, 485, 2087), (35, 486, 2087), (34, 487, 2088), (34, 488, 2089), (34, 489, 2089), (34, 490, 2090), (34, 491, 2090), (34, 492, 2090), (34, 493, 2091), (33, 494, 2092), (33, 495, 2093), (33, 496, 2093), (33, 497, 2094), (33, 498, 2094), (33, 499, 2094), (33, 500, 2095), (32, 501, 2096), (32, 502, 2097), (32, 503, 2097), (32, 504, 2098), (32, 505, 2098), (32, 506, 2099), (32, 507, 2099), (31, 508, 2101), (31, 509, 2101), (31, 510, 2102), (31, 511, 2102), (31, 512, 2103), (31, 513, 2103), (31, 514, 2104), (31, 515, 2104), (30, 516, 2106), (30, 517, 2107), (30, 518, 2107), (30, 519, 2108), (30, 520, 2108), (30, 521, 2109), (30, 522, 2109), (30, 523, 2110), (29, 524, 2112), (29, 525, 2112), (29, 526, 2113), (29, 527, 2114), (29, 528, 2114), (29, 529, 2115), (29, 530, 2116), (29, 531, 2116), (28, 532, 2118), (28, 533, 2119), (28, 534, 2120), (28, 535, 2120), (28, 536, 2121), (28, 537, 2122), (28, 538, 2122), (28, 539, 2123), (28, 540, 2123), (28, 541, 2123), (27, 542, 2124), (27, 543, 2125), (27, 544, 2125), (27, 545, 2125), (27, 546, 2126), (27, 547, 2126), (27, 548, 2126), (27, 549, 2127), (27, 550, 2127), (27, 551, 2127), (27, 552, 2127), (27, 553, 2128), (27, 554, 2128), (27, 555, 2128), (27, 556, 2129), (27, 557, 2129), (27, 558, 2129), (27, 559, 2129), (27, 560, 2130), (26, 561, 2131), (26, 562, 2131), (26, 563, 2131), (26, 564, 2132), (26, 565, 2132), (26, 566, 2132), (26, 567, 2132), (26, 568, 2133), (26, 569, 2133), (26, 570, 2133), (26, 571, 2133), (26, 572, 2134), (26, 573, 2134), (26, 574, 2134), (26, 575, 2134), (26, 576, 2135), (26, 577, 2135), (26, 578, 2135), (26, 579, 2135), (26, 580, 2136), (25, 581, 2137), (25, 582, 2137), (25, 583, 2137), (25, 584, 2138), (25, 585, 2138), (25, 586, 2138), (25, 587, 2138), (25, 588, 2138), (25, 589, 2139), (25, 590, 2139), (25, 591, 2139), (25, 592, 2139), (25, 593, 2140), (25, 594, 2140), (25, 595, 2140), (25, 596, 2140), (25, 597, 2140), (25, 598, 2141), (25, 599, 2141), (25, 600, 2141), (25, 601, 2141), (24, 602, 2142), (24, 603, 2143), (24, 604, 2143), (24, 605, 2143), (24, 606, 2143), (24, 607, 2143), (24, 608, 2144), (24, 609, 2144), (24, 610, 2144), (24, 611, 2144), (24, 612, 2144), (24, 613, 2145), (24, 614, 2145), (24, 615, 2145), (24, 616, 2145), (24, 617, 2145), (24, 618, 2145), (24, 619, 2145), (24, 620, 2145), (24, 621, 2145), (24, 622, 2145), (24, 623, 2145), (23, 624, 2146), (23, 625, 2146), (23, 626, 2146), (23, 627, 2146), (23, 628, 2146), (23, 629, 2147), (23, 630, 2147), (23, 631, 2147), (23, 632, 2147), (23, 633, 2147), (23, 634, 2147), (23, 635, 2147), (23, 636, 2147), (23, 637, 2147), (23, 638, 2147), (23, 639, 2147), (23, 640, 2147), (23, 641, 2147), (23, 642, 2147), (23, 643, 2147), (23, 644, 2147), (23, 645, 2147), (23, 646, 2148), (23, 647, 2148), (22, 648, 2149), (22, 649, 2149), (22, 650, 2149), (22, 651, 2149), (22, 652, 2149), (22, 653, 2149), (22, 654, 2149), (22, 655, 2149), (22, 656, 2149), (22, 657, 2149), (22, 658, 2149), (22, 659, 2149), (22, 660, 2149), (22, 661, 2149), (22, 662, 2149), (22, 663, 2149), (22, 664, 2150), (22, 665, 2150), (22, 666, 2150), (22, 667, 2150), (22, 668, 2150), (22, 669, 2150), (22, 670, 2150), (22, 671, 2150), (21, 672, 2151), (21, 673, 2151), (21, 674, 2151), (21, 675, 2151), (21, 676, 2151), (21, 677, 2151), (21, 678, 2151), (21, 679, 2151), (21, 680, 2151), (21, 681, 2151), (21, 682, 2151), (21, 683, 2152), (21, 684, 2152), (21, 685, 2152), (21, 686, 2152), (21, 687, 2152), (21, 688, 2152), (21, 689, 2152), (21, 690, 2152), (21, 691, 2152), (21, 692, 2152), (21, 693, 2152), (21, 694, 2152), (21, 695, 2152), (21, 696, 2152), (21, 697, 2151), (21, 698, 2151), (22, 699, 2150), (22, 700, 2150), (22, 701, 2150), (22, 702, 2150), (22, 703, 2150), (22, 704, 2150), (22, 705, 2150), (22, 706, 2150), (22, 707, 2150), (22, 708, 2150), (22, 709, 2150), (22, 710, 2150), (22, 711, 2150), (23, 712, 2149), (23, 713, 2149), (23, 714, 2149), (23, 715, 2149), (23, 716, 2149), (23, 717, 2149), (23, 718, 2149), (23, 719, 2148), (23, 720, 2148), (23, 721, 2148), (23, 722, 2148), (23, 723, 2148), (24, 724, 2147), (24, 725, 2147), (24, 726, 2147), (24, 727, 2147), (24, 728, 2147), (24, 729, 2147), (24, 730, 2147), (24, 731, 2147), (24, 732, 2147), (24, 733, 2147), (24, 734, 2147), (24, 735, 2147), (25, 736, 2146), (25, 737, 2146), (25, 738, 2146), (25, 739, 2145), (25, 740, 2145), (25, 741, 2145), (25, 742, 2145), (25, 743, 2145), (25, 744, 2145), (25, 745, 2145), (25, 746, 2145), (25, 747, 2145), (26, 748, 2144), (26, 749, 2144), (26, 750, 2144), (26, 751, 2144), (26, 752, 2144), (26, 753, 2144), (26, 754, 2144), (26, 755, 2144), (26, 756, 2144), (26, 757, 2144), (26, 758, 2144), (26, 759, 2143), (27, 760, 2142), (27, 761, 2142), (27, 762, 2142), (27, 763, 2142), (27, 764, 2142), (27, 765, 2142), (27, 766, 2142), (27, 767, 2142), (27, 768, 2142), (27, 769, 2142), (27, 770, 2142), (27, 771, 2142), (27, 772, 2142), (27, 773, 2142), (27, 774, 2142), (27, 775, 2142), (27, 776, 2142), (27, 777, 2142), (27, 778, 2141), (27, 779, 2141), (27, 780, 2141), (27, 781, 2141), (27, 782, 2141), (27, 783, 2141), (27, 784, 2141), (27, 785, 2141), (27, 786, 2141), (27, 787, 2141), (27, 788, 2141), (27, 789, 2141), (27, 790, 2141), (27, 791, 2141), (27, 792, 2141), (27, 793, 2141), (27, 794, 2141), (27, 795, 2141), (26, 796, 2142), (26, 797, 2141), (26, 798, 2141), (26, 799, 2141), (26, 800, 2141), (26, 801, 2141), (26, 802, 2141), (26, 803, 2141), (26, 804, 2141), (26, 805, 2141), (26, 806, 2141), (26, 807, 2141), (26, 808, 2141), (26, 809, 2141), (26, 810, 2141), (26, 811, 2141), (26, 812, 2141), (26, 813, 2141), (26, 814, 2140), (26, 815, 2140), (26, 816, 2140), (26, 817, 2140), (26, 818, 2140), (26, 819, 2140), (26, 820, 2140), (26, 821, 2140), (26, 822, 2140), (26, 823, 2140), (26, 824, 2140), (26, 825, 2140), (26, 826, 2140), (26, 827, 2140), (26, 828, 2140), (26, 829, 2140), (26, 830, 2140), (26, 831, 2139), (26, 832, 2139), (26, 833, 2139), (26, 834, 2139), (26, 835, 2139), (26, 836, 2139), (26, 837, 2139), (26, 838, 2139), (26, 839, 2139), (26, 840, 2139), (26, 841, 2139), (26, 842, 2138), (26, 843, 2138), (26, 844, 2138), (26, 845, 2137), (26, 846, 2137), (26, 847, 2136), (26, 848, 2136), (26, 849, 2136), (26, 850, 2135), (26, 851, 2135), (26, 852, 2134), (26, 853, 2134), (26, 854, 2133), (26, 855, 2133), (27, 856, 2132), (27, 857, 2131), (27, 858, 2131), (27, 859, 2130), (27, 860, 2130), (27, 861, 2129), (27, 862, 2129), (27, 863, 2128), (27, 864, 2128), (27, 865, 2127), (27, 866, 2127), (27, 867, 2126), (27, 868, 2126), (27, 869, 2125), (27, 870, 2125), (27, 871, 2124), (27, 872, 2124), (27, 873, 2123), (27, 874, 2123), (28, 875, 2121), (28, 876, 2120), (28, 877, 2120), (28, 878, 2119), (28, 879, 2119), (28, 880, 2118), (28, 881, 2117), (28, 882, 2117), (28, 883, 2116), (28, 884, 2116), (28, 885, 2115), (28, 886, 2115), (28, 887, 2114), (28, 888, 2113), (28, 889, 2113), (28, 890, 2112), (28, 891, 2112), (28, 892, 2111), (28, 893, 2111), (29, 894, 2109), (29, 895, 2109), (29, 896, 2108), (29, 897, 2108), (29, 898, 2107), (29, 899, 2107), (29, 900, 2107), (29, 901, 2106), (29, 902, 2106), (29, 903, 2105), (29, 904, 2105), (29, 905, 2104), (29, 906, 2104), (29, 907, 2104), (29, 908, 2103), (29, 909, 2103), (29, 910, 2102), (29, 911, 2102), (30, 912, 2100), (30, 913, 2100), (30, 914, 2100), (30, 915, 2099), (30, 916, 2099), (30, 917, 2099), (30, 918, 2098), (30, 919, 2098), (30, 920, 2098), (30, 921, 2098), (30, 922, 2097), (30, 923, 2097), (29, 924, 2098), (29, 925, 2098), (29, 926, 2097), (29, 927, 2097), (29, 928, 2097), (29, 929, 2097), (29, 930, 2096), (29, 931, 2096), (29, 932, 2096), (29, 933, 2096), (29, 934, 2096), (29, 935, 2095), (29, 936, 2095), (29, 937, 2095), (29, 938, 2095), (29, 939, 2094), (29, 940, 2094), (29, 941, 2094), (29, 942, 2094), (29, 943, 2094), (29, 944, 2093), (29, 945, 2093), (29, 946, 2093), (29, 947, 2093), (29, 948, 2092), (29, 949, 2092), (29, 950, 2092), (29, 951, 2092), (29, 952, 2092), (28, 953, 2092), (28, 954, 2092), (28, 955, 2092), (28, 956, 2092), (28, 957, 2092), (28, 958, 2091), (28, 959, 2091), (28, 960, 2091), (28, 961, 2091), (28, 962, 2091), (28, 963, 2090), (28, 964, 2090), (28, 965, 2090), (28, 966, 2090), (28, 967, 2090), (28, 968, 2090), (28, 969, 2089), (28, 970, 2089), (28, 971, 2089), (28, 972, 2089), (28, 973, 2089), (28, 974, 2088), (28, 975, 2088), (28, 976, 2088), (28, 977, 2088), (28, 978, 2088), (28, 979, 2088), (28, 980, 2087), (28, 981, 2087), (28, 982, 2087), (27, 983, 2088), (27, 984, 2088), (27, 985, 2088), (27, 986, 2087), (27, 987, 2087), (27, 988, 2087), (27, 989, 2087), (27, 990, 2087), (27, 991, 2087), (27, 992, 2086), (27, 993, 2086), (27, 994, 2086), (27, 995, 2085), (27, 996, 2085), (27, 997, 2085), (27, 998, 2084), (27, 999, 2084), (27, 1000, 2084), (27, 1001, 2083), (28, 1002, 2082), (28, 1003, 2082), (28, 1004, 2081), (28, 1005, 2081), (28, 1006, 2081), (28, 1007, 2080), (28, 1008, 2080), (28, 1009, 2080), (28, 1010, 2079), (28, 1011, 2079), (28, 1012, 2079), (28, 1013, 2078), (28, 1014, 2078), (28, 1015, 2078), (28, 1016, 2077), (28, 1017, 2077), (28, 1018, 2076), (28, 1019, 2076), (28, 1020, 2076), (28, 1021, 2075), (28, 1022, 2075), (28, 1023, 2075), (28, 1024, 2074), (28, 1025, 2074), (28, 1026, 2073), (28, 1027, 2073), (28, 1028, 2073), (28, 1029, 2072), (28, 1030, 2072), (29, 1031, 2070), (29, 1032, 2070), (29, 1033, 2069), (29, 1034, 2069), (29, 1035, 2069), (29, 1036, 2068), (29, 1037, 2068), (29, 1038, 2067), (29, 1039, 2067), (29, 1040, 2066), (29, 1041, 2066), (29, 1042, 2066), (29, 1043, 2065), (29, 1044, 2065), (29, 1045, 2064), (29, 1046, 2064), (29, 1047, 2063), (29, 1048, 2063), (29, 1049, 2062), (29, 1050, 2062), (29, 1051, 2061), (29, 1052, 2061), (29, 1053, 2060), (29, 1054, 2060), (29, 1055, 2059), (29, 1056, 2059), (29, 1057, 2058), (29, 1058, 2058), (30, 1059, 2056), (30, 1060, 2056), (30, 1061, 2055), (30, 1062, 2055), (30, 1063, 2054), (30, 1064, 2054), (30, 1065, 2053), (30, 1066, 2052), (30, 1067, 2052), (30, 1068, 2051), (30, 1069, 2051), (30, 1070, 2050), (30, 1071, 2049), (30, 1072, 2048), (30, 1073, 2047), (30, 1074, 2047), (30, 1075, 2046), (30, 1076, 2045), (30, 1077, 2044), (30, 1078, 2043), (30, 1079, 2042), (30, 1080, 2041), (30, 1081, 2040), (30, 1082, 2039), (30, 1083, 2038), (30, 1084, 2037), (29, 1085, 2037), (29, 1086, 2036), (29, 1087, 2035), (29, 1088, 2034), (29, 1089, 2033), (29, 1090, 2032), (29, 1091, 2031), (29, 1092, 2030), (29, 1093, 2029), (29, 1094, 2028), (29, 1095, 2027), (29, 1096, 2026), (29, 1097, 2026), (29, 1098, 2025), (29, 1099, 2024), (29, 1100, 2023), (29, 1101, 2022), (29, 1102, 2021), (29, 1103, 2021), (29, 1104, 2020), (29, 1105, 2019), (29, 1106, 2019), (29, 1107, 2018), (29, 1108, 2017), (29, 1109, 2016), (29, 1110, 2016), (29, 1111, 2015), (29, 1112, 2014), (29, 1113, 2014), (29, 1114, 2013), (29, 1115, 2013), (29, 1116, 2012), (29, 1117, 2011), (29, 1118, 2011), (29, 1119, 2010), (29, 1120, 2010), (29, 1121, 2009), (29, 1122, 2009), (29, 1123, 2008), (29, 1124, 2008), (29, 1125, 2007), (29, 1126, 2006), (29, 1127, 2006), (29, 1128, 2005), (29, 1129, 2005), (29, 1130, 2004), (29, 1131, 2004), (29, 1132, 2004), (29, 1133, 2003), (29, 1134, 2003), (29, 1135, 2002), (29, 1136, 2002), (29, 1137, 2001), (28, 1138, 2002), (28, 1139, 2001), (28, 1140, 2001), (28, 1141, 2001), (28, 1142, 2000), (28, 1143, 2000), (28, 1144, 2000), (28, 1145, 1999), (28, 1146, 1999), (28, 1147, 1999), (28, 1148, 1998), (28, 1149, 1998), (28, 1150, 1998), (28, 1151, 1997), (29, 1152, 1996), (29, 1153, 1996), (29, 1154, 1995), (29, 1155, 1995), (29, 1156, 1995), (29, 1157, 1994), (29, 1158, 1994), (29, 1159, 1994), (29, 1160, 1993), (29, 1161, 1993), (29, 1162, 1992), (29, 1163, 1992), (29, 1164, 1992), (29, 1165, 1991), (29, 1166, 1991), (29, 1167, 1991), (29, 1168, 1990), (29, 1169, 1990), (29, 1170, 1989), (29, 1171, 1989), (29, 1172, 1989), (29, 1173, 1988), (29, 1174, 1988), (29, 1175, 1987), (29, 1176, 1987), (29, 1177, 1987), (29, 1178, 1986), (29, 1179, 1986), (29, 1180, 1985), (29, 1181, 1985), (29, 1182, 1985), (29, 1183, 1984), (29, 1184, 1984), (29, 1185, 1983), (29, 1186, 1983), (29, 1187, 1982), (29, 1188, 1982), (29, 1189, 1981), (29, 1190, 1981), (29, 1191, 1981), (29, 1192, 1980), (29, 1193, 1980), (29, 1194, 1979), (29, 1195, 1979), (29, 1196, 1978), (29, 1197, 1978), (29, 1198, 1977), (29, 1199, 1977), (29, 1200, 1976), (29, 1201, 1976), (29, 1202, 1975), (29, 1203, 1975), (29, 1204, 1974), (29, 1205, 1973), (29, 1206, 1973), (29, 1207, 1972), (29, 1208, 1972), (29, 1209, 1971), (29, 1210, 1971), (29, 1211, 1970), (29, 1212, 1970), (29, 1213, 1969), (29, 1214, 1968), (29, 1215, 1968), (29, 1216, 1967), (29, 1217, 1967), (29, 1218, 1966), (29, 1219, 1965), (29, 1220, 1965), (29, 1221, 1964), (29, 1222, 1963), (29, 1223, 1963), (29, 1224, 1962), (29, 1225, 1961), (29, 1226, 1960), (29, 1227, 1960), (29, 1228, 1959), (29, 1229, 1958), (29, 1230, 1957), (29, 1231, 1956), (29, 1232, 1955), (29, 1233, 1955), (29, 1234, 1954), (29, 1235, 1953), (29, 1236, 1952), (29, 1237, 1951), (29, 1238, 1951), (30, 1239, 1949), (30, 1240, 1948), (30, 1241, 1947), (30, 1242, 1947), (30, 1243, 1946), (30, 1244, 1945), (30, 1245, 1945), (30, 1246, 1944), (30, 1247, 1943), (30, 1248, 1943), (30, 1249, 1942), (30, 1250, 1942), (30, 1251, 1941), (30, 1252, 1940), (30, 1253, 1940), (30, 1254, 1939), (30, 1255, 1939), (30, 1256, 1938), (30, 1257, 1938), (30, 1258, 1937), (30, 1259, 1936), (30, 1260, 1936), (30, 1261, 1935), (30, 1262, 1935), (30, 1263, 1934), (30, 1264, 1934), (30, 1265, 1933), (30, 1266, 1933), (30, 1267, 1932), (30, 1268, 1932), (30, 1269, 1932), (30, 1270, 1931), (30, 1271, 1931), (30, 1272, 1930), (30, 1273, 1930), (30, 1274, 1929), (30, 1275, 1929), (30, 1276, 1928), (30, 1277, 1928), (30, 1278, 1928), (30, 1279, 1927), (30, 1280, 1927), (30, 1281, 1926), (30, 1282, 1926), (30, 1283, 1926), (30, 1284, 1925), (30, 1285, 1925), (30, 1286, 1925), (30, 1287, 1924), (30, 1288, 1924), (30, 1289, 1924), (30, 1290, 1923), (30, 1291, 1923), (30, 1292, 1922), (30, 1293, 1922), (30, 1294, 1922), (30, 1295, 1922), (30, 1296, 1921), (30, 1297, 1921), (30, 1298, 1921), (30, 1299, 1921), (30, 1300, 1920), (30, 1301, 1920), (30, 1302, 1920), (30, 1303, 1920), (30, 1304, 1919), (30, 1305, 1919), (30, 1306, 1919), (30, 1307, 1919), (30, 1308, 1918), (30, 1309, 1918), (30, 1310, 1918), (30, 1311, 1918), (31, 1312, 1916), (31, 1313, 1916), (31, 1314, 1916), (31, 1315, 1916), (31, 1316, 1915), (31, 1317, 1915), (31, 1318, 1915), (31, 1319, 1915), (31, 1320, 1914), (31, 1321, 1914), (31, 1322, 1914), (31, 1323, 1914), (31, 1324, 1913), (31, 1325, 1913), (31, 1326, 1913), (31, 1327, 1913), (31, 1328, 1912), (31, 1329, 1912), (31, 1330, 1912), (31, 1331, 1912), (31, 1332, 1911), (31, 1333, 1911), (31, 1334, 1911), (31, 1335, 1910), (31, 1336, 1910), (31, 1337, 1910), (31, 1338, 1910), (32, 1339, 1908), (32, 1340, 1908), (32, 1341, 1908), (32, 1342, 1907), (32, 1343, 1907), (32, 1344, 1907), (32, 1345, 1907), (32, 1346, 1906), (32, 1347, 1906), (32, 1348, 1906), (32, 1349, 1905), (32, 1350, 1905), (32, 1351, 1905), (32, 1352, 1905), (32, 1353, 1904), (32, 1354, 1904), (32, 1355, 1904), (32, 1356, 1903), (32, 1357, 1903), (32, 1358, 1903), (32, 1359, 1902), (32, 1360, 1902), (32, 1361, 1902), (32, 1362, 1901), (32, 1363, 1901), (32, 1364, 1901), (32, 1365, 1901), (33, 1366, 1899), (33, 1367, 1899), (33, 1368, 1899), (33, 1369, 1898), (33, 1370, 1898), (33, 1371, 1897), (33, 1372, 1897), (33, 1373, 1896), (33, 1374, 1896), (33, 1375, 1895), (33, 1376, 1895), (33, 1377, 1894), (33, 1378, 1894), (33, 1379, 1893), (33, 1380, 1893), (33, 1381, 1892), (34, 1382, 1891), (34, 1383, 1890), (34, 1384, 1889), (34, 1385, 1889), (34, 1386, 1888), (34, 1387, 1888), (34, 1388, 1887), (34, 1389, 1886), (34, 1390, 1886), (34, 1391, 1885), (34, 1392, 1884), (34, 1393, 1883), (34, 1394, 1883), (34, 1395, 1882), (35, 1396, 1880), (35, 1397, 1880), (35, 1398, 1879), (35, 1399, 1878), (35, 1400, 1877), (35, 1401, 1876), (35, 1402, 1876), (35, 1403, 1875), (35, 1404, 1874), (35, 1405, 1873), (35, 1406, 1872), (35, 1407, 1871), (35, 1408, 1870), (35, 1409, 1869), (36, 1410, 1867), (36, 1411, 1866), (36, 1412, 1866), (36, 1413, 1865), (36, 1414, 1864), (36, 1415, 1863), (36, 1416, 1862), (36, 1417, 1861), (36, 1418, 1861), (36, 1419, 1860), (36, 1420, 1859), (36, 1421, 1858), (36, 1422, 1858), (37, 1423, 1856), (37, 1424, 1855), (37, 1425, 1854), (37, 1426, 1854), (37, 1427, 1853), (37, 1428, 1852), (37, 1429, 1852), (37, 1430, 1851), (37, 1431, 1851), (37, 1432, 1850), (37, 1433, 1849), (37, 1434, 1849), (38, 1435, 1847), (38, 1436, 1847), (38, 1437, 1846), (38, 1438, 1846), (38, 1439, 1845), (38, 1440, 1844), (38, 1441, 1844), (38, 1442, 1843), (38, 1443, 1843), (38, 1444, 1842), (38, 1445, 1842), (38, 1446, 1841), (38, 1447, 1841), (38, 1448, 1841), (38, 1449, 1841), (38, 1450, 1841), (39, 1451, 1839), (39, 1452, 1839), (39, 1453, 1839), (39, 1454, 1839), (39, 1455, 1838), (39, 1456, 1838), (39, 1457, 1838), (39, 1458, 1838), (39, 1459, 1838), (39, 1460, 1837), (39, 1461, 1837), (39, 1462, 1837), (39, 1463, 1837), (39, 1464, 1837), (39, 1465, 1836), (39, 1466, 1836), (39, 1467, 1836), (39, 1468, 1836), (39, 1469, 1836), (39, 1470, 1835), (39, 1471, 1835), (39, 1472, 1835), (39, 1473, 1835), (39, 1474, 1835), (39, 1475, 1834), (39, 1476, 1834), (39, 1477, 1834), (39, 1478, 1834), (39, 1479, 1834), (39, 1480, 1833), (39, 1481, 1833), (39, 1482, 1833), (39, 1483, 1833), (39, 1484, 1833), (39, 1485, 1833), (39, 1486, 1832), (39, 1487, 1832), (39, 1488, 1832), (39, 1489, 1832), (39, 1490, 1832), (39, 1491, 1831), (39, 1492, 1831), (39, 1493, 1831), (40, 1494, 1830), (40, 1495, 1830), (40, 1496, 1830), (40, 1497, 1829), (40, 1498, 1829), (40, 1499, 1829), (40, 1500, 1829), (40, 1501, 1829), (40, 1502, 1829), (40, 1503, 1828), (40, 1504, 1828), (40, 1505, 1828), (40, 1506, 1828), (40, 1507, 1828), (40, 1508, 1828), (40, 1509, 1827), (40, 1510, 1827), (40, 1511, 1827), (40, 1512, 1827), (40, 1513, 1827), (40, 1514, 1827), (40, 1515, 1826), (40, 1516, 1826), (40, 1517, 1826), (40, 1518, 1826), (40, 1519, 1826), (40, 1520, 1826), (40, 1521, 1825), (40, 1522, 1825), (40, 1523, 1825), (40, 1524, 1825), (40, 1525, 1825), (40, 1526, 1825), (40, 1527, 1825), (40, 1528, 1825), (40, 1529, 1824), (40, 1530, 1824), (40, 1531, 1824), (40, 1532, 1824), (40, 1533, 1824), (40, 1534, 1824), (40, 1535, 1824), (40, 1536, 1824), (41, 1537, 1822), (41, 1538, 1822), (41, 1539, 1822), (41, 1540, 1822), (41, 1541, 1822), (41, 1542, 1822), (41, 1543, 1822), (41, 1544, 1822), (41, 1545, 1821), (41, 1546, 1821), (41, 1547, 1821), (41, 1548, 1821), (41, 1549, 1821), (41, 1550, 1821), (41, 1551, 1821), (41, 1552, 1821), (41, 1553, 1820), (41, 1554, 1820), (41, 1555, 1820), (41, 1556, 1820), (41, 1557, 1820), (41, 1558, 1820), (41, 1559, 1820), (41, 1560, 1820), (41, 1561, 1819), (41, 1562, 1819), (41, 1563, 1819), (41, 1564, 1819), (41, 1565, 1819), (41, 1566, 1819), (41, 1567, 1819), (41, 1568, 1819), (41, 1569, 1818), (41, 1570, 1818), (41, 1571, 1818), (41, 1572, 1818), (41, 1573, 1818), (41, 1574, 1818), (41, 1575, 1818), (41, 1576, 1818), (41, 1577, 1817), (41, 1578, 1817), (41, 1579, 1817), (41, 1580, 1817), (41, 1581, 1817), (42, 1582, 1816), (42, 1583, 1816), (42, 1584, 1816), (42, 1585, 1815), (42, 1586, 1815), (42, 1587, 1815), (42, 1588, 1815), (42, 1589, 1815), (42, 1590, 1815), (42, 1591, 1815), (42, 1592, 1814), (42, 1593, 1814), (42, 1594, 1814), (42, 1595, 1814), (42, 1596, 1814), (42, 1597, 1814), (42, 1598, 1814), (42, 1599, 1814), (42, 1600, 1813), (42, 1601, 1813), (42, 1602, 1813), (42, 1603, 1813), (42, 1604, 1813), (42, 1605, 1813), (41, 1606, 1814), (41, 1607, 1814), (41, 1608, 1814), (41, 1609, 1813), (41, 1610, 1813), (41, 1611, 1813), (41, 1612, 1813), (41, 1613, 1813), (41, 1614, 1813), (41, 1615, 1813), (41, 1616, 1813), (41, 1617, 1813), (41, 1618, 1812), (41, 1619, 1812), (41, 1620, 1812), (41, 1621, 1812), (41, 1622, 1812), (41, 1623, 1812), (41, 1624, 1812), (41, 1625, 1812), (41, 1626, 1811), (41, 1627, 1811), (41, 1628, 1811), (40, 1629, 1812), (40, 1630, 1812), (40, 1631, 1812), (40, 1632, 1812), (40, 1633, 1812), (40, 1634, 1811), (40, 1635, 1811), (40, 1636, 1811), (40, 1637, 1811), (40, 1638, 1811), (40, 1639, 1811), (40, 1640, 1811), (40, 1641, 1811), (40, 1642, 1810), (40, 1643, 1810), (40, 1644, 1810), (40, 1645, 1810), (40, 1646, 1810), (40, 1647, 1810), (40, 1648, 1810), (40, 1649, 1809), (40, 1650, 1809), (40, 1651, 1809), (40, 1652, 1809), (39, 1653, 1810), (39, 1654, 1810), (39, 1655, 1810), (39, 1656, 1810), (39, 1657, 1809), (39, 1658, 1809), (39, 1659, 1809), (39, 1660, 1809), (39, 1661, 1809), (39, 1662, 1809), (39, 1663, 1809), (39, 1664, 1808), (39, 1665, 1808), (39, 1666, 1808), (39, 1667, 1808), (39, 1668, 1808), (39, 1669, 1808), (39, 1670, 1808), (39, 1671, 1807), (39, 1672, 1807), (39, 1673, 1807), (39, 1674, 1807), (39, 1675, 1807), (39, 1676, 1806), (39, 1677, 1806), (39, 1678, 1806), (40, 1679, 1805), (40, 1680, 1804), (40, 1681, 1804), (40, 1682, 1804), (40, 1683, 1804), (40, 1684, 1803), (41, 1685, 1802), (41, 1686, 1802), (41, 1687, 1801), (41, 1688, 1801), (41, 1689, 1801), (41, 1690, 1801), (42, 1691, 1799), (42, 1692, 1799), (42, 1693, 1799), (42, 1694, 1799), (42, 1695, 1798), (42, 1696, 1798), (43, 1697, 1797), (43, 1698, 1796), (43, 1699, 1796), (43, 1700, 1796), (43, 1701, 1795), (43, 1702, 1795), (44, 1703, 1794), (44, 1704, 1794), (44, 1705, 1793), (44, 1706, 1793), (44, 1707, 1793), (45, 1708, 1791), (45, 1709, 1791), (45, 1710, 1791), (45, 1711, 1790), (45, 1712, 1790), (45, 1713, 1790), (46, 1714, 1788), (46, 1715, 1788), (46, 1716, 1788), (46, 1717, 1787), (46, 1718, 1787), (47, 1719, 1786), (47, 1720, 1785), (47, 1721, 1785), (47, 1722, 1785), (47, 1723, 1784), (47, 1724, 1784), (48, 1725, 1782), (48, 1726, 1782), (48, 1727, 1782), (48, 1728, 1781), (49, 1729, 1780), (49, 1730, 1780), (49, 1731, 1779), (49, 1732, 1779), (49, 1733, 1778), (50, 1734, 1777), (50, 1735, 1777), (50, 1736, 1776), (50, 1737, 1776), (50, 1738, 1775), (51, 1739, 1774), (51, 1740, 1773), (51, 1741, 1773), (51, 1742, 1773), (52, 1743, 1771), (52, 1744, 1771), (52, 1745, 1770), (52, 1746, 1770), (52, 1747, 1769), (53, 1748, 1768), (53, 1749, 1767), (53, 1750, 1767), (53, 1751, 1767), (53, 1752, 1766), (53, 1753, 1766), (53, 1754, 1765), (53, 1755, 1765), (53, 1756, 1765), (53, 1757, 1764), (53, 1758, 1764), (53, 1759, 1764), (53, 1760, 1763), (53, 1761, 1763), (53, 1762, 1762), (53, 1763, 1762), (53, 1764, 1762), (53, 1765, 1761), (53, 1766, 1761), (53, 1767, 1761), (53, 1768, 1760), (53, 1769, 1760), (53, 1770, 1760), (53, 1771, 1759), (53, 1772, 1759), (53, 1773, 1758), (53, 1774, 1758), (53, 1775, 1758), (53, 1776, 1757), (53, 1777, 1757), (53, 1778, 1757), (53, 1779, 1756), (53, 1780, 1756), (53, 1781, 1756), (53, 1782, 1755), (53, 1783, 1755), (53, 1784, 1755), (53, 1785, 1754), (53, 1786, 1754), (53, 1787, 1754), (53, 1788, 1754), (53, 1789, 1753), (53, 1790, 1753), (53, 1791, 1753), (53, 1792, 1752), (53, 1793, 1752), (53, 1794, 1752), (53, 1795, 1751), (53, 1796, 1751), (53, 1797, 1751), (53, 1798, 1750), (53, 1799, 1750), (53, 1800, 1750), (53, 1801, 1750), (54, 1802, 1748), (54, 1803, 1748), (54, 1804, 1748), (54, 1805, 1747), (54, 1806, 1747), (54, 1807, 1747), (54, 1808, 1747), (54, 1809, 1746), (54, 1810, 1746), (54, 1811, 1746), (54, 1812, 1745), (54, 1813, 1745), (54, 1814, 1745), (54, 1815, 1745), (54, 1816, 1744), (54, 1817, 1744), (54, 1818, 1744), (54, 1819, 1744), (54, 1820, 1743), (54, 1821, 1743), (54, 1822, 1743), (54, 1823, 1743), (54, 1824, 1742), (54, 1825, 1742), (55, 1826, 1741), (55, 1827, 1740), (56, 1828, 1739), (56, 1829, 1739), (57, 1830, 1737), (57, 1831, 1737), (58, 1832, 1736), (58, 1833, 1735), (59, 1834, 1734), (59, 1835, 1734), (60, 1836, 1732), (60, 1837, 1732), (60, 1838, 1732), (61, 1839, 1730), (61, 1840, 1730), (62, 1841, 1729), (62, 1842, 1728), (63, 1843, 1727), (63, 1844, 1727), (64, 1845, 1725), (64, 1846, 1725), (65, 1847, 1724), (65, 1848, 1723), (66, 1849, 1722), (66, 1850, 1721), (66, 1851, 1721), (67, 1852, 1720), (67, 1853, 1719), (68, 1854, 1718), (68, 1855, 1718), (69, 1856, 1716), (69, 1857, 1716), (70, 1858, 1714), (70, 1859, 1714), (71, 1860, 1713), (71, 1861, 1712), (71, 1862, 1712), (72, 1863, 1711), (72, 1864, 1710), (73, 1865, 1709), (73, 1866, 1708), (74, 1867, 1707), (74, 1868, 1707), (75, 1869, 1705), (75, 1870, 1705), (75, 1871, 1704), (76, 1872, 1703), (76, 1873, 1703), (77, 1874, 1701), (77, 1875, 1701), (78, 1876, 1699), (78, 1877, 1699), (78, 1878, 1699), (79, 1879, 1697), (79, 1880, 1697), (80, 1881, 1695), (80, 1882, 1695), (81, 1883, 1693), (81, 1884, 1693), (81, 1885, 1693), (82, 1886, 1691), (82, 1887, 1691), (83, 1888, 1689), (83, 1889, 1689), (84, 1890, 1687), (84, 1891, 1687), (84, 1892, 1687), (85, 1893, 1685), (85, 1894, 1685), (86, 1895, 1683), (86, 1896, 1683), (87, 1897, 1681), (87, 1898, 1681), (87, 1899, 1680), (88, 1900, 1679), (88, 1901, 1678), (88, 1902, 1678), (89, 1903, 1677), (89, 1904, 1676), (90, 1905, 1675), (90, 1906, 1674), (90, 1907, 1674), (91, 1908, 1672), (91, 1909, 1672), (92, 1910, 1670), (92, 1911, 1669), (92, 1912, 1669), (93, 1913, 1667), (93, 1914, 1667), (94, 1915, 1665), (94, 1916, 1665), (94, 1917, 1664), (95, 1918, 1663), (95, 1919, 1662), (96, 1920, 1661), (96, 1921, 1660), (97, 1922, 1658), (97, 1923, 1658), (97, 1924, 1657), (98, 1925, 1656), (98, 1926, 1655), (99, 1927, 1653), (99, 1928, 1653), (100, 1929, 1651), (100, 1930, 1651), (101, 1931, 1649), (101, 1932, 1648), (102, 1933, 1647), (102, 1934, 1646), (103, 1935, 1644), (103, 1936, 1644), (104, 1937, 1642), (105, 1938, 1640), (105, 1939, 1640), (106, 1940, 1638), (106, 1941, 1637), (107, 1942, 1635), (107, 1943, 1635), (108, 1944, 1633), (109, 1945, 1631), (109, 1946, 1630), (110, 1947, 1628), (110, 1948, 1628), (111, 1949, 1626), (112, 1950, 1624), (112, 1951, 1623), (113, 1952, 1622), (114, 1953, 1620), (114, 1954, 1619), (115, 1955, 1617), (116, 1956, 1615), (116, 1957, 1615), (117, 1958, 1613), (118, 1959, 1611), (119, 1960, 1610), (119, 1961, 1609), (120, 1962, 1607), (121, 1963, 87), (209, 1963, 1517), (122, 1964, 74), (216, 1964, 1510), (123, 1965, 62), (223, 1965, 1502), (123, 1966, 51), (230, 1966, 1494), (124, 1967, 40), (237, 1967, 1487), (125, 1968, 31), (244, 1968, 1479), (126, 1969, 21), (252, 1969, 1470), (127, 1970, 12), (259, 1970, 1462), (128, 1971, 4), (266, 1971, 1455), (274, 1972, 1446), (282, 1973, 1437), (290, 1974, 1429), (294, 1975, 1424), (299, 1976, 1418), (304, 1977, 1412), (309, 1978, 1406), (314, 1979, 1400), (319, 1980, 1394), (325, 1981, 1387), (331, 1982, 1380), (337, 1983, 1373), (344, 1984, 1365), (351, 1985, 1357), (358, 1986, 1349), (366, 1987, 1340), (372, 1988, 1332), (376, 1989, 1327), (380, 1990, 1322), (384, 1991, 1317), (388, 1992, 1311), (392, 1993, 1306), (397, 1994, 1300), (401, 1995, 1295), (406, 1996, 1288), (410, 1997, 1283), (415, 1998, 1276), (420, 1999, 1270), (424, 2000, 1265), (429, 2001, 1258), (434, 2002, 1252), (439, 2003, 1245), (445, 2004, 1238), (450, 2005, 1231), (455, 2006, 1224), (460, 2007, 1218), (465, 2008, 1211), (470, 2009, 1204), (475, 2010, 1198), (480, 2011, 1191), (486, 2012, 1183), (491, 2013, 1176), (496, 2014, 1169), (502, 2015, 1161), (507, 2016, 1089), (513, 2017, 1065), (519, 2018, 1055), (524, 2019, 1046), (530, 2020, 1036), (535, 2021, 1027), (540, 2022, 1018), (545, 2023, 1009), (550, 2024, 1000), (555, 2025, 992), (560, 2026, 983), (564, 2027, 975), (569, 2028, 967), (573, 2029, 959), (578, 2030, 951), (582, 2031, 943), (587, 2032, 935), (591, 2033, 928), (595, 2034, 920), (599, 2035, 913), (603, 2036, 906), (607, 2037, 899), (611, 2038, 892), (614, 2039, 883), (617, 2040, 869), (619, 2041, 857), (622, 2042, 844), (624, 2043, 832), (627, 2044, 820), (629, 2045, 808), (631, 2046, 797), (634, 2047, 786), (636, 2048, 780), (638, 2049, 775), (640, 2050, 770), (642, 2051, 764), (644, 2052, 760), (645, 2053, 756), (647, 2054, 751), (649, 2055, 746), (651, 2056, 740), (652, 2057, 736), (654, 2058, 731), (656, 2059, 726), (658, 2060, 720), (660, 2061, 715), (662, 2062, 709), (664, 2063, 704), (666, 2064, 698), (669, 2065, 691), (671, 2066, 685), (673, 2067, 679), (675, 2068, 673), (678, 2069, 666), (680, 2070, 660), (683, 2071, 652), (685, 2072, 645), (688, 2073, 636), (691, 2074, 628), (694, 2075, 620), (698, 2076, 610), (703, 2077, 599), (707, 2078, 590), (712, 2079, 579), (716, 2080, 569), (721, 2081, 558), (725, 2082, 548), (729, 2083, 538), (734, 2084, 527), (738, 2085, 517), (742, 2086, 508), (747, 2087, 497), (751, 2088, 488), (755, 2089, 479), (759, 2090, 470), (763, 2091, 461), (768, 2092, 452), (772, 2093, 443), (775, 2094, 436), (779, 2095, 427), (782, 2096, 420), (785, 2097, 412), (789, 2098, 404), (792, 2099, 397), (795, 2100, 390), (798, 2101, 383), (801, 2102, 376), (805, 2103, 370), (808, 2104, 365), (811, 2105, 359), (814, 2106, 354), (817, 2107, 349), (821, 2108, 342), (824, 2109, 337), (827, 2110, 332), (830, 2111, 327), (833, 2112, 322), (836, 2113, 317), (839, 2114, 312), (842, 2115, 307), (845, 2116, 302), (848, 2117, 297), (851, 2118, 293), (854, 2119, 288), (857, 2120, 283), (860, 2121, 278), (863, 2122, 274), (866, 2123, 269), (869, 2124, 264), (872, 2125, 260), (875, 2126, 255), (877, 2127, 251), (880, 2128, 247), (883, 2129, 242), (886, 2130, 237), (889, 2131, 232), (893, 2132, 226), (896, 2133, 221), (899, 2134, 216), (902, 2135, 211), (906, 2136, 204), (909, 2137, 199), (913, 2138, 193), (916, 2139, 188), (920, 2140, 181), (924, 2141, 175), (928, 2142, 167), (932, 2143, 155), (935, 2144, 144), (945, 2145, 126), (955, 2146, 108), (966, 2147, 89), (977, 2148, 69), (988, 2149, 50), (999, 2150, 30), (1011, 2151, 9)], ['935,2144,771,2092,675,2068,610,2037,371,1987,222,1964,128,1971,54,1825,53,1748,39,1678,39,1451,29,1238,26,759,21,672,38,462,92,312,117,279,206,209,286,181,370,133,523,124,1426,124,1575,129,1664,141,1787,184,1904,206,2014,298,2081,385,2150,539,2168,613,2171,718,2164,841,2129,912,2113,991,2080,1069,2042,1112,2009,1191,1957,1276,1931,1368,1878,1446,1846,1670,1782,1863,1718,1974,1662,2015,1573,2018,1496,2039,1420,2046,1329,2072,1177,2101,1094,2142'])], 'temp/1741131394_2449776_917877156_a9c2d4b99270c9302def4ed40606e685.jpg']} nb pixel non reg : 3692295 nb pixel common : 3685546 proportion of common points : 0.998172139550063 [('test release memory', 'SUCCESS', True), ('test detect objet', 'SUCCESS', True), ('test polygone', 'SUCCESS', True)] res_total : True #&_# TEST SUCCEEDED #&_# : tests/mask_test #&_# /home/admin/workarea/git/Velours/python/tests/python_tests.py refs/heads/master_7b90246f7285e983362b27a932a2f1f627b3aad9 SQL :INSERT INTO MTRAdmin.monitor_sys (name, type, server, version_code, result_str, result_bool, lien , test_group ,test_name) VALUES ('python_test3','1','marlene','refs/heads/master_7b90246f7285e983362b27a932a2f1f627b3aad9','{"mask_detection": "success"}','1','http://marlene.fotonower-preprod.com/job/2025/March/05032025/python_test3//data_2/data_log/job/2025/March/05032025/python_test3/log-python3----short_python3--v--marlene-00:35:01.txt','mask_detection','unknown'); #&_# END OF TEST #&_# : tests/mask_test #&_# #&_# BEGIN OF TEST : tests/datou_test #&_# /home/admin/workarea/git/Velours/python/tests/datou_test.py Datou All Test python version used : 3 ############################### TEST sam ################################ TEST SAM Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4573 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4573 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4573 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4573 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! no param json to modify List Step Type Loaded in datou : sam list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1189321094) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 1189321094 download finish for photo 1189321094 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.18898820877075195 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! WARNING : we have an input that is not a photo, we should get rid of it Calling datou_exec Inside datou_exec : verbose : True number of steps : 1 step1:sam Wed Mar 5 00:37:31 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1741131451_2449776_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png': 1189321094} map_photo_id_path_extension : {1189321094: {'path': 'temp/1741131451_2449776_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png', 'extension': 'png'}} map_subphoto_mainphoto : {} Beginning of datou step sam ! pht : 4677 Inside sam : nb paths : 1 (640, 960, 3) time for calcul the mask position with numpy : 0.0019767284393310547 nb_pixel_total : 5626 time to create 1 rle with old method : 0.0067327022552490234 time for calcul the mask position with numpy : 0.001361846923828125 nb_pixel_total : 11876 time to create 1 rle with old method : 0.013074874877929688 time for calcul the mask position with numpy : 0.0016851425170898438 nb_pixel_total : 83771 time to create 1 rle with old method : 0.09007763862609863 time for calcul the mask position with numpy : 0.001356363296508789 nb_pixel_total : 3777 time to create 1 rle with old method : 0.004215717315673828 time for calcul the mask position with numpy : 0.0014190673828125 nb_pixel_total : 16429 time to create 1 rle with old method : 0.01779627799987793 time for calcul the mask position with numpy : 0.0013918876647949219 nb_pixel_total : 14600 time to create 1 rle with old method : 0.01616191864013672 time for calcul the mask position with numpy : 0.001398324966430664 nb_pixel_total : 3770 time to create 1 rle with old method : 0.004215717315673828 time for calcul the mask position with numpy : 0.0013339519500732422 nb_pixel_total : 5513 time to create 1 rle with old method : 0.0059888362884521484 time for calcul the mask position with numpy : 0.0014863014221191406 nb_pixel_total : 10802 time to create 1 rle with old method : 0.01215219497680664 time for calcul the mask position with numpy : 0.0013897418975830078 nb_pixel_total : 7590 time to create 1 rle with old method : 0.008417367935180664 time for calcul the mask position with numpy : 0.0014119148254394531 nb_pixel_total : 5948 time to create 1 rle with old method : 0.006617307662963867 time for calcul the mask position with numpy : 0.001522064208984375 nb_pixel_total : 29508 time to create 1 rle with old method : 0.03195524215698242 time for calcul the mask position with numpy : 0.0013306140899658203 nb_pixel_total : 3352 time to create 1 rle with old method : 0.003911733627319336 time for calcul the mask position with numpy : 0.0013842582702636719 nb_pixel_total : 2937 time to create 1 rle with old method : 0.0034551620483398438 time for calcul the mask position with numpy : 0.0014827251434326172 nb_pixel_total : 13936 time to create 1 rle with old method : 0.015701770782470703 time for calcul the mask position with numpy : 0.0014672279357910156 nb_pixel_total : 9893 time to create 1 rle with old method : 0.011196136474609375 time for calcul the mask position with numpy : 0.0013570785522460938 nb_pixel_total : 9504 time to create 1 rle with old method : 0.010413169860839844 time for calcul the mask position with numpy : 0.0013163089752197266 nb_pixel_total : 4273 time to create 1 rle with old method : 0.004538059234619141 time for calcul the mask position with numpy : 0.0012984275817871094 nb_pixel_total : 1222 time to create 1 rle with old method : 0.0014796257019042969 time for calcul the mask position with numpy : 0.0014486312866210938 nb_pixel_total : 3970 time to create 1 rle with old method : 0.004410505294799805 time for calcul the mask position with numpy : 0.0013248920440673828 nb_pixel_total : 6636 time to create 1 rle with old method : 0.007805824279785156 time for calcul the mask position with numpy : 0.001316070556640625 nb_pixel_total : 2377 time to create 1 rle with old method : 0.0028328895568847656 time for calcul the mask position with numpy : 0.001420736312866211 nb_pixel_total : 1515 time to create 1 rle with old method : 0.0018236637115478516 time for calcul the mask position with numpy : 0.0013842582702636719 nb_pixel_total : 16344 time to create 1 rle with old method : 0.019956588745117188 time for calcul the mask position with numpy : 0.001497507095336914 nb_pixel_total : 2077 time to create 1 rle with old method : 0.0024123191833496094 time for calcul the mask position with numpy : 0.0014643669128417969 nb_pixel_total : 2421 time to create 1 rle with old method : 0.0029726028442382812 time for calcul the mask position with numpy : 0.0014102458953857422 nb_pixel_total : 693 time to create 1 rle with old method : 0.0009257793426513672 time for calcul the mask position with numpy : 0.0014216899871826172 nb_pixel_total : 10609 time to create 1 rle with old method : 0.012386322021484375 time for calcul the mask position with numpy : 0.0015082359313964844 nb_pixel_total : 13043 time to create 1 rle with old method : 0.015355825424194336 time for calcul the mask position with numpy : 0.0014102458953857422 nb_pixel_total : 4300 time to create 1 rle with old method : 0.0053653717041015625 time for calcul the mask position with numpy : 0.0014190673828125 nb_pixel_total : 2382 time to create 1 rle with old method : 0.002901315689086914 time for calcul the mask position with numpy : 0.0016086101531982422 nb_pixel_total : 38951 time to create 1 rle with old method : 0.04304051399230957 time for calcul the mask position with numpy : 0.0014233589172363281 nb_pixel_total : 1489 time to create 1 rle with old method : 0.0017006397247314453 time for calcul the mask position with numpy : 0.001444101333618164 nb_pixel_total : 8640 time to create 1 rle with old method : 0.009727239608764648 time for calcul the mask position with numpy : 0.0014302730560302734 nb_pixel_total : 3531 time to create 1 rle with old method : 0.004133462905883789 time for calcul the mask position with numpy : 0.0014171600341796875 nb_pixel_total : 2446 time to create 1 rle with old method : 0.002769947052001953 time for calcul the mask position with numpy : 0.0013170242309570312 nb_pixel_total : 2792 time to create 1 rle with old method : 0.0031261444091796875 time for calcul the mask position with numpy : 0.0013561248779296875 nb_pixel_total : 2781 time to create 1 rle with old method : 0.0033342838287353516 time for calcul the mask position with numpy : 0.0013997554779052734 nb_pixel_total : 5375 time to create 1 rle with old method : 0.00623321533203125 time for calcul the mask position with numpy : 0.0013058185577392578 nb_pixel_total : 1250 time to create 1 rle with old method : 0.0014636516571044922 time for calcul the mask position with numpy : 0.0013234615325927734 nb_pixel_total : 1025 time to create 1 rle with old method : 0.0011813640594482422 time for calcul the mask position with numpy : 0.001371145248413086 nb_pixel_total : 1647 time to create 1 rle with old method : 0.001878499984741211 time for calcul the mask position with numpy : 0.0013113021850585938 nb_pixel_total : 4134 time to create 1 rle with old method : 0.0048787593841552734 time for calcul the mask position with numpy : 0.0013184547424316406 nb_pixel_total : 346 time to create 1 rle with old method : 0.00046443939208984375 time for calcul the mask position with numpy : 0.0013651847839355469 nb_pixel_total : 4193 time to create 1 rle with old method : 0.004855155944824219 time for calcul the mask position with numpy : 0.00151824951171875 nb_pixel_total : 27898 time to create 1 rle with old method : 0.030765295028686523 time for calcul the mask position with numpy : 0.0012984275817871094 nb_pixel_total : 2028 time to create 1 rle with old method : 0.00234222412109375 time for calcul the mask position with numpy : 0.0014557838439941406 nb_pixel_total : 18433 time to create 1 rle with old method : 0.020842313766479492 time for calcul the mask position with numpy : 0.0014717578887939453 nb_pixel_total : 858 time to create 1 rle with old method : 0.0011734962463378906 time for calcul the mask position with numpy : 0.0014240741729736328 nb_pixel_total : 1566 time to create 1 rle with old method : 0.0020253658294677734 time for calcul the mask position with numpy : 0.0014226436614990234 nb_pixel_total : 594 time to create 1 rle with old method : 0.0008234977722167969 time for calcul the mask position with numpy : 0.0014226436614990234 nb_pixel_total : 894 time to create 1 rle with old method : 0.0012187957763671875 time for calcul the mask position with numpy : 0.0014867782592773438 nb_pixel_total : 1061 time to create 1 rle with old method : 0.0013880729675292969 time for calcul the mask position with numpy : 0.001415252685546875 nb_pixel_total : 221 time to create 1 rle with old method : 0.00033092498779296875 time for calcul the mask position with numpy : 0.0014412403106689453 nb_pixel_total : 2333 time to create 1 rle with old method : 0.0028285980224609375 time for calcul the mask position with numpy : 0.0014376640319824219 nb_pixel_total : 887 time to create 1 rle with old method : 0.0011603832244873047 time for calcul the mask position with numpy : 0.0014586448669433594 nb_pixel_total : 337 time to create 1 rle with old method : 0.0004711151123046875 time for calcul the mask position with numpy : 0.0014286041259765625 nb_pixel_total : 2765 time to create 1 rle with old method : 0.0033359527587890625 time for calcul the mask position with numpy : 0.001420736312866211 nb_pixel_total : 492 time to create 1 rle with old method : 0.0006916522979736328 time for calcul the mask position with numpy : 0.0014693737030029297 nb_pixel_total : 585 time to create 1 rle with old method : 0.0011258125305175781 time for calcul the mask position with numpy : 0.0018014907836914062 nb_pixel_total : 581 time to create 1 rle with old method : 0.001138448715209961 time for calcul the mask position with numpy : 0.001689910888671875 nb_pixel_total : 1208 time to create 1 rle with old method : 0.0022399425506591797 time for calcul the mask position with numpy : 0.0015931129455566406 nb_pixel_total : 16754 time to create 1 rle with old method : 0.02101421356201172 time for calcul the mask position with numpy : 0.0014545917510986328 nb_pixel_total : 1695 time to create 1 rle with old method : 0.0020618438720703125 time for calcul the mask position with numpy : 0.0014302730560302734 nb_pixel_total : 1075 time to create 1 rle with old method : 0.0012540817260742188 time for calcul the mask position with numpy : 0.0014023780822753906 nb_pixel_total : 1442 time to create 1 rle with old method : 0.00186920166015625 time for calcul the mask position with numpy : 0.0013740062713623047 nb_pixel_total : 8594 time to create 1 rle with old method : 0.009961366653442383 time for calcul the mask position with numpy : 0.0014615058898925781 nb_pixel_total : 8512 time to create 1 rle with old method : 0.009300708770751953 time for calcul the mask position with numpy : 0.0014476776123046875 nb_pixel_total : 836 time to create 1 rle with old method : 0.001041412353515625 time for calcul the mask position with numpy : 0.0012967586517333984 nb_pixel_total : 3098 time to create 1 rle with old method : 0.0036344528198242188 time for calcul the mask position with numpy : 0.0014503002166748047 nb_pixel_total : 1842 time to create 1 rle with old method : 0.0023260116577148438 time for calcul the mask position with numpy : 0.0012960433959960938 nb_pixel_total : 1423 time to create 1 rle with old method : 0.0017096996307373047 time for calcul the mask position with numpy : 0.0012888908386230469 nb_pixel_total : 974 time to create 1 rle with old method : 0.0011827945709228516 time for calcul the mask position with numpy : 0.0013823509216308594 nb_pixel_total : 1745 time to create 1 rle with old method : 0.0022106170654296875 time for calcul the mask position with numpy : 0.001440286636352539 nb_pixel_total : 9106 time to create 1 rle with old method : 0.010739803314208984 time for calcul the mask position with numpy : 0.001348257064819336 nb_pixel_total : 3218 time to create 1 rle with old method : 0.0036003589630126953 time for calcul the mask position with numpy : 0.0012819766998291016 nb_pixel_total : 267 time to create 1 rle with old method : 0.0003159046173095703 time for calcul the mask position with numpy : 0.0013165473937988281 nb_pixel_total : 1344 time to create 1 rle with old method : 0.0017800331115722656 time for calcul the mask position with numpy : 0.0014491081237792969 nb_pixel_total : 970 time to create 1 rle with old method : 0.0012156963348388672 time for calcul the mask position with numpy : 0.0014531612396240234 nb_pixel_total : 7536 time to create 1 rle with old method : 0.008621454238891602 time for calcul the mask position with numpy : 0.001382589340209961 nb_pixel_total : 711 time to create 1 rle with old method : 0.0009469985961914062 time for calcul the mask position with numpy : 0.0013737678527832031 nb_pixel_total : 969 time to create 1 rle with old method : 0.0012865066528320312 time for calcul the mask position with numpy : 0.001363515853881836 nb_pixel_total : 248 time to create 1 rle with old method : 0.00032401084899902344 time for calcul the mask position with numpy : 0.0013892650604248047 nb_pixel_total : 5093 time to create 1 rle with old method : 0.006103038787841797 time for calcul the mask position with numpy : 0.0014410018920898438 nb_pixel_total : 1642 time to create 1 rle with old method : 0.0021257400512695312 time for calcul the mask position with numpy : 0.0014371871948242188 nb_pixel_total : 735 time to create 1 rle with old method : 0.0010597705841064453 time for calcul the mask position with numpy : 0.0014348030090332031 nb_pixel_total : 300 time to create 1 rle with old method : 0.00042247772216796875 time for calcul the mask position with numpy : 0.001369476318359375 nb_pixel_total : 2377 time to create 1 rle with old method : 0.0029511451721191406 time for calcul the mask position with numpy : 0.001439809799194336 nb_pixel_total : 1124 time to create 1 rle with old method : 0.001415252685546875 time for calcul the mask position with numpy : 0.001428365707397461 nb_pixel_total : 596 time to create 1 rle with old method : 0.0008003711700439453 time for calcul the mask position with numpy : 0.001451253890991211 nb_pixel_total : 914 time to create 1 rle with old method : 0.001241445541381836 time for calcul the mask position with numpy : 0.0013995170593261719 nb_pixel_total : 890 time to create 1 rle with old method : 0.0011324882507324219 time for calcul the mask position with numpy : 0.0013515949249267578 nb_pixel_total : 2200 time to create 1 rle with old method : 0.002712726593017578 time for calcul the mask position with numpy : 0.0014255046844482422 nb_pixel_total : 886 time to create 1 rle with old method : 0.0011529922485351562 time for calcul the mask position with numpy : 0.0014433860778808594 nb_pixel_total : 947 time to create 1 rle with old method : 0.0012214183807373047 time for calcul the mask position with numpy : 0.0013957023620605469 nb_pixel_total : 1320 time to create 1 rle with old method : 0.00170135498046875 time for calcul the mask position with numpy : 0.0014395713806152344 nb_pixel_total : 828 time to create 1 rle with old method : 0.0011038780212402344 time for calcul the mask position with numpy : 0.0014677047729492188 nb_pixel_total : 1609 time to create 1 rle with old method : 0.001997232437133789 insert ignore into MTRPhoto.crop_hashtag_ids (photo_id, hashtag_id, `type`,x0,x1,y0,y1,score) VALUES (%s,%s,%s,%s,%s,%s,%s,%s) batch 1 Loaded 98 chid ids of type : 4677 Number RLEs to save : 8761 INSERT IGNORE INTO MTRPhoto.crop_segments (`crop_hashtag_id`, `x0`, `y0`, `length`) VALUES (%s, %s, %s , %s) first line : ('3693825957', '464', '201', '4') ... last line : ('3693826054', '280', '238', '18') INSERT IGNORE INTO MTRPhoto.crop_sum_segments (`crop_hashtag_id`, `sum_segments`) VALUES (%s, %s) TO DO : save crop sub photo not yet done ! After datou_step_exec type output : map_portfolio_photo : len 0 keys : dict_keys([]) Inside saveOutput : final : True verbose : True saveOutput not yet implemented for datou_step.type : sam we use saveGeneral [1189321094] map_info['map_portfolio_photo'] : {} final : True mtd_id 4573 list_pids : [1189321094] Looping around the photos to save general results len do output : 1 /1189321094Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4573', None, None, None, None, None, None, None, None) ('4573', None, '1189321094', None, None, None, None, None, None) begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4573', None, '1189321094', 'None', None, None, None, None, None)] time used for this insertion : 0.5401883125305176 save_final save missing photos in datou_result : time spend for datou_step_exec : 10.061349630355835 time spend to save output : 0.540529727935791 total time spend for step 1 : 10.601879358291626 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {'1189321094': [[, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ], 'temp/1741131451_2449776_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png']} nb_objects detect : 98 ############################### TEST frcnn ################################ test frcnn Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4184 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4184 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4184 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4184 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! no param json to modify List Step Type Loaded in datou : frcnn list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (917754606) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 917754606 download finish for photo 917754606 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.19281935691833496 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 1 step1:frcnn Wed Mar 5 00:37:42 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1741131462_2449776_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg': 917754606} map_photo_id_path_extension : {917754606: {'path': 'temp/1741131462_2449776_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou step Faster rcnn ! classes : ['background', 'plaque'] pht : 4370 caffemodel_name (should be vgg16_immat_307 but not used because net loaded outside in the fonction) : {'id': 3375, 'mtr_user_id': 31, 'name': 'detection_plaque_valcor_010622', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,plaque', 'svm_portfolios_learning': '0,0', 'photo_hashtag_type': 4370, 'photo_desc_type': 5676, 'type_classification': 'caffe_faster_rcnn', 'hashtag_id_list': '0,0'} To loadFromThcl() model_param file didn't exist model_name : detection_plaque_valcor_010622 model_type : caffe_faster_rcnn list file need : ['caffemodel', 'test.prototxt'] file exist in s3 : ['caffemodel', 'test.prototxt'] file manque in s3 : [] WARNING: Logging before InitGoogleLogging() is written to STDERR F0305 00:37:44.676657 2449776 syncedmem.cpp:71] Check failed: error == cudaSuccess (2 vs. 0) out of memory *** Check failure stack trace: *** Command terminated by signal 6 38.08user 25.40system 2:19.52elapsed 45%CPU (0avgtext+0avgdata 3601676maxresident)k 3061584inputs+4640outputs (5607major+3115329minor)pagefaults 0swaps