python /home/admin/mtr/script_for_cron.py -j python_test3 -m 12 -a ' --short_python3 -v ' -s python_test3 -M 0 -S 0 -U 100,100,120 import MySQLdb succeeded Import error (python version) python version = 3 warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data list_job_run_as_list : ['mask_detection', 'datou', 'CacheModelData_queries', 'CachePhotoData_queries', 'test_fork', 'prepare_maskdata', 'portfolio_queries', 'sla_mensuel'] python version used : 3 liste_fichiers : [('tests/mask_test', True, 'Test mask-detection ', 'mask_detection'), ('tests/datou_test', True, 'Datou All Test', 'datou', 'all'), ('mtr/database_queries/CacheModelData_queries', True, 'Test Cache Model Data', 'CacheModelData_queries'), ('tests/cache_photo_data_test', True, 'Test local_cache_photo ', 'CachePhotoData_queries'), ('mtr/mask_rcnn/prepare_maskdata', True, 'test prepare mask data', 'prepare_maskdata', 'all'), ('mtr/database_queries/portfolio_queries', True, 'test portfolio queries', 'portfolio_queries'), ('prod/memo/memo', True, 'SLA Mensuel', 'sla_mensuel', 'all')] #&_# BEGIN OF TEST : tests/mask_test #&_# /home/admin/workarea/git/Velours/python/tests/mask_test.py Test mask-detection python version used : 3 ############################### TEST memory used ################################ free memory at begining : begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5525 run mask_detect Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.17614173889160156 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:mask_detect Mon Mar 17 21:35:28 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5525 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 /home/admin/workarea/git/Velours/python/tests/python_tests.py:11: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp 2025-03-17 21:35:32.210161: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-17 21:35:32.239208: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-17 21:35:32.240999: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fc640000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-17 21:35:32.241062: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-17 21:35:32.245048: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-17 21:35:32.489359: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xc323500 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-17 21:35:32.489399: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-17 21:35:32.490546: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-17 21:35:32.492616: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-17 21:35:32.520930: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-17 21:35:32.537448: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-17 21:35:32.541034: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-17 21:35:32.569444: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-17 21:35:32.573524: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-17 21:35:32.623166: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-17 21:35:32.624394: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-17 21:35:32.624853: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-17 21:35:32.626247: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-17 21:35:32.626263: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-17 21:35:32.626271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-17 21:35:32.627623: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5051 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl454 thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3473 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21)) {'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_coco_origin NUM_CLASSES 81 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 2025-03-17 21:35:33.263052: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-17 21:35:33.263175: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-17 21:35:33.263194: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-17 21:35:33.263212: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-17 21:35:33.263228: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-17 21:35:33.263243: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-17 21:35:33.263273: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-17 21:35:33.263289: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-17 21:35:33.264211: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-17 21:35:33.265390: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-17 21:35:33.265422: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-17 21:35:33.265436: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-17 21:35:33.265449: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-17 21:35:33.265463: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-17 21:35:33.265475: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-17 21:35:33.265488: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-17 21:35:33.265500: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-17 21:35:33.266353: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-17 21:35:33.266392: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-17 21:35:33.266400: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-17 21:35:33.266407: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-17 21:35:33.267330: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5051 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : mask_coco_origin model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-03-17 21:35:41.633102: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-17 21:35:41.987227: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-17 21:35:43.882874: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 21:35:44.649267: E tensorflow/stream_executor/cuda/cuda_driver.cc:910] failed to synchronize the stop event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-03-17 21:35:44.649305: E tensorflow/stream_executor/gpu/gpu_timer.cc:55] Internal: Error destroying CUDA event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-03-17 21:35:44.649316: E tensorflow/stream_executor/gpu/gpu_timer.cc:60] Internal: Error destroying CUDA event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-03-17 21:35:44.649338: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 8B (8 bytes) from device: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-03-17 21:35:44.649351: E tensorflow/stream_executor/stream.cc:5485] Internal: Failed to enqueue async memset operation: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-03-17 21:35:44.649365: W tensorflow/core/kernels/gpu_utils.cc:69] Failed to check cudnn convolutions for out-of-bounds reads and writes with an error message: 'Failed to load in-memory CUBIN: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered'; skipping this check. This only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-03-17 21:35:44.649376: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 8B (8 bytes) from device: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 2025-03-17 21:35:44.649389: I tensorflow/stream_executor/stream.cc:4963] [stream=0xd2a45b0,impl=0xd2a35a0] did not memzero GPU location; source: 0x7fc4ae7fa020 2025-03-17 21:35:44.649673: F ./tensorflow/core/kernels/reduction_gpu_kernels.cu.h:731] Non-OK-status: GpuLaunchKernel(RowReduceKernel, num_blocks, threads_per_block, 0, cu_stream, in, out, num_rows, num_cols, op, init) status: Internal: an illegal memory access was encountered max_time_sub_proc : 3600 Useless call to update_current_state in case -12 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : False eke 12-6-18 : saveMask need to be cleaned for new output ! ERROR : mask output needs to be a dictionnary now ! No output to save, continue without doing anything ! save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : -12 free memory after detection : begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5525 ############################### TEST detect object ################################ run mask_detect Inside batchDatouExec : verbose : False Catched exception ! Connect or reconnect ! # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.20363807678222656 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:mask_detect Mon Mar 17 22:35:31 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5525 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-03-17 22:35:36.095042: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-17 22:35:36.131141: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-17 22:35:36.133452: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fc640000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-17 22:35:36.133483: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-17 22:35:36.139168: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-17 22:35:36.373863: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xbd786b0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-17 22:35:36.373936: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-17 22:35:36.375475: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-17 22:35:36.375937: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-17 22:35:36.378530: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-17 22:35:36.384542: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-17 22:35:36.385114: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-17 22:35:36.398249: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-17 22:35:36.400983: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-17 22:35:36.436678: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-17 22:35:36.438217: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-17 22:35:36.438732: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-17 22:35:36.439533: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-17 22:35:36.439555: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-17 22:35:36.439567: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-17 22:35:36.441112: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5051 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl454 thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3473 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21)) {'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_coco_origin NUM_CLASSES 81 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 2025-03-17 22:35:37.071886: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-17 22:35:37.071997: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-17 22:35:37.072021: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-17 22:35:37.072042: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-17 22:35:37.072063: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-17 22:35:37.072083: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-17 22:35:37.072104: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-17 22:35:37.072124: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-17 22:35:37.073358: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-17 22:35:37.074474: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-17 22:35:37.074519: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-17 22:35:37.074543: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-17 22:35:37.074565: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-17 22:35:37.074587: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-17 22:35:37.074608: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-17 22:35:37.074629: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-17 22:35:37.074651: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-17 22:35:37.075902: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-17 22:35:37.075936: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-17 22:35:37.075947: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-17 22:35:37.075957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-17 22:35:37.077232: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5051 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : mask_coco_origin model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-03-17 22:35:45.777824: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-17 22:35:45.973716: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-17 22:35:47.912443: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:47.913246: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.60G (3865470464 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:48.798930: W tensorflow/core/common_runtime/bfc_allocator.cc:311] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature. 2025-03-17 22:35:48.880307: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:48.880387: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-17 22:35:48.881804: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:48.881834: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-17 22:35:48.892272: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:48.892319: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.78GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-17 22:35:48.893289: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:48.893309: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.78GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-17 22:35:48.923724: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:48.923813: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 19.91MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-17 22:35:48.923851: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-03-17 22:35:48.925318: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:48.925347: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 16.00MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-17 22:35:48.926799: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:48.926833: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 16.00MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-17 22:35:48.935563: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:48.935623: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 63.85MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-17 22:35:48.936419: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:48.936437: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 63.85MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-17 22:35:48.937235: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:48.937252: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.26GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-17 22:35:48.937978: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:48.949185: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:48.949958: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:48.969604: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:48.970669: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:48.971694: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:48.972606: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:48.977700: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:48.978440: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:48.979196: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:48.979987: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:48.981235: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:48.992987: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:48.993733: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:49.005406: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:49.006149: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:49.006913: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:49.007658: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:49.008429: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:35:49.009219: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/mask_coco_origin /data/models_weight/mask_coco_origin/mask_model.h5 size_local : 257557808 size in s3 : 257557808 create time local : 2021-08-09 05:27:17 create time in s3 : 2021-08-06 19:45:17 mask_model.h5 already exist and didn't need to update list_images length : 1 NEW PHOTO Processing 1 images image shape: (720, 1280, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 89) min: 0.00000 max: 1280.00000 nb d'objets trouves : 4 Detection mask done ! Trying to reset tf kernel 889013 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 3533 tf kernel not reseted sub process len(results) : 1 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 1 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 4726 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl454 Catched exception ! Connect or reconnect ! thcls : [{'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3473 ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] time for calcul the mask position with numpy : 0.0004267692565917969 nb_pixel_total : 16902 time to create 1 rle with old method : 0.020656347274780273 length of segment : 107 time for calcul the mask position with numpy : 0.11672067642211914 nb_pixel_total : 480737 time to create 1 rle with new method : 0.026249408721923828 length of segment : 632 time for calcul the mask position with numpy : 0.00040650367736816406 nb_pixel_total : 36587 time to create 1 rle with old method : 0.04329943656921387 length of segment : 132 time for calcul the mask position with numpy : 8.797645568847656e-05 nb_pixel_total : 4794 time to create 1 rle with old method : 0.0060749053955078125 length of segment : 51 time spent for convertir_results : 1.1514575481414795 time spend for datou_step_exec : 22.30762815475464 time spend to save output : 4.2438507080078125e-05 total time spend for step 1 : 22.30767059326172 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : False eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 403 chid ids of type : 445 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Number RLEs to save : 0 begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 time used for this insertion : 0.013835668563842773 save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {'917855882': [[(917855882, 492601069, 445, 1092, 1280, 0, 108, 0.9988373, [(1205, 1, 58), (1165, 2, 105), (1159, 3, 113), (1149, 4, 124), (1113, 5, 161), (1100, 6, 174), (1097, 7, 177), (1095, 8, 179), (1095, 9, 179), (1095, 10, 179), (1095, 11, 179), (1095, 12, 179), (1095, 13, 179), (1095, 14, 178), (1095, 15, 178), (1095, 16, 178), (1095, 17, 178), (1095, 18, 177), (1095, 19, 177), (1095, 20, 177), (1095, 21, 177), (1095, 22, 177), (1095, 23, 178), (1095, 24, 178), (1095, 25, 178), (1095, 26, 179), (1095, 27, 179), (1095, 28, 180), (1095, 29, 181), (1095, 30, 182), (1095, 31, 183), (1095, 32, 183), (1095, 33, 184), (1095, 34, 184), (1096, 35, 183), (1096, 36, 183), (1096, 37, 184), (1097, 38, 183), (1097, 39, 183), (1097, 40, 183), (1098, 41, 182), (1098, 42, 182), (1098, 43, 182), (1099, 44, 181), (1099, 45, 181), (1099, 46, 181), (1100, 47, 180), (1100, 48, 180), (1101, 49, 179), (1101, 50, 179), (1102, 51, 178), (1102, 52, 178), (1103, 53, 177), (1103, 54, 177), (1104, 55, 176), (1104, 56, 176), (1104, 57, 176), (1104, 58, 176), (1105, 59, 175), (1105, 60, 175), (1105, 61, 175), (1105, 62, 175), (1105, 63, 175), (1106, 64, 174), (1106, 65, 174), (1106, 66, 174), (1106, 67, 174), (1106, 68, 174), (1106, 69, 174), (1106, 70, 174), (1106, 71, 174), (1106, 72, 174), (1106, 73, 174), (1107, 74, 173), (1107, 75, 173), (1107, 76, 173), (1107, 77, 173), (1107, 78, 173), (1107, 79, 173), (1108, 80, 172), (1108, 81, 172), (1109, 82, 171), (1110, 83, 170), (1110, 84, 170), (1111, 85, 169), (1112, 86, 168), (1113, 87, 166), (1114, 88, 165), (1115, 89, 164), (1117, 90, 162), (1120, 91, 159), (1138, 92, 141), (1146, 93, 133), (1154, 94, 125), (1167, 95, 112), (1177, 96, 102), (1183, 97, 95), (1185, 98, 93), (1187, 99, 90), (1188, 100, 55), (1264, 100, 12), (1190, 101, 50), (1191, 102, 46), (1194, 103, 40), (1197, 104, 34), (1202, 105, 25), (1207, 106, 16)], ['1222,106,1207,106,1206,105,1197,104,1191,102,1182,96,1176,95,1167,95,1166,94,1154,94,1153,93,1146,93,1145,92,1137,91,1120,91,1115,89,1110,84,1107,79,1106,73,1106,64,1104,55,1099,46,1095,34,1095,8,1100,6,1112,6,1113,5,1148,5,1149,4,1158,4,1165,2,1204,2,1205,1,1262,1,1269,2,1273,5,1273,13,1271,18,1271,22,1273,27,1277,31,1279,37,1279,86,1278,87,1278,96,1275,100,1264,100,1263,99,1243,99,1230,104']), (917855882, 492601069, 445, 52, 1128, 16, 668, 0.99775, [(711, 22, 21), (926, 22, 46), (608, 23, 146), (894, 23, 103), (598, 24, 234), (850, 24, 158), (590, 25, 427), (582, 26, 444), (575, 27, 458), (569, 28, 466), (565, 29, 472), (560, 30, 480), (556, 31, 486), (550, 32, 495), (544, 33, 503), (538, 34, 512), (532, 35, 520), (527, 36, 527), (523, 37, 534), (518, 38, 541), (514, 39, 548), (510, 40, 554), (506, 41, 561), (503, 42, 566), (499, 43, 572), (496, 44, 577), (493, 45, 582), (491, 46, 585), (488, 47, 590), (487, 48, 592), (485, 49, 595), (483, 50, 598), (482, 51, 600), (481, 52, 602), (480, 53, 603), (479, 54, 605), (478, 55, 606), (476, 56, 608), (475, 57, 610), (474, 58, 611), (473, 59, 613), (472, 60, 614), (470, 61, 616), (469, 62, 618), (468, 63, 619), (466, 64, 621), (465, 65, 623), (464, 66, 624), (462, 67, 626), (461, 68, 628), (459, 69, 630), (458, 70, 631), (456, 71, 633), (455, 72, 635), (453, 73, 637), (452, 74, 638), (451, 75, 639), (449, 76, 641), (448, 77, 642), (447, 78, 643), (446, 79, 644), (445, 80, 645), (444, 81, 646), (442, 82, 648), (441, 83, 649), (440, 84, 650), (439, 85, 651), (438, 86, 652), (437, 87, 653), (436, 88, 654), (435, 89, 655), (434, 90, 656), (433, 91, 657), (432, 92, 658), (431, 93, 659), (430, 94, 660), (429, 95, 661), (428, 96, 662), (427, 97, 663), (425, 98, 665), (423, 99, 667), (421, 100, 669), (419, 101, 671), (417, 102, 673), (413, 103, 677), (410, 104, 680), (405, 105, 685), (401, 106, 689), (397, 107, 693), (392, 108, 698), (387, 109, 703), (382, 110, 708), (377, 111, 713), (373, 112, 717), (369, 113, 721), (365, 114, 725), (362, 115, 728), (358, 116, 732), (356, 117, 734), (353, 118, 737), (351, 119, 739), (349, 120, 741), (346, 121, 744), (344, 122, 746), (341, 123, 749), (338, 124, 752), (335, 125, 755), (331, 126, 759), (327, 127, 763), (323, 128, 767), (319, 129, 770), (314, 130, 775), (308, 131, 781), (303, 132, 786), (294, 133, 795), (287, 134, 802), (279, 135, 810), (273, 136, 816), (267, 137, 822), (262, 138, 827), (258, 139, 831), (255, 140, 834), (252, 141, 837), (250, 142, 839), (247, 143, 842), (245, 144, 844), (242, 145, 847), (240, 146, 849), (237, 147, 852), (234, 148, 855), (230, 149, 859), (226, 150, 863), (220, 151, 869), (213, 152, 876), (207, 153, 882), (200, 154, 889), (193, 155, 896), (187, 156, 902), (184, 157, 905), (181, 158, 908), (178, 159, 911), (176, 160, 913), (174, 161, 915), (172, 162, 917), (170, 163, 919), (168, 164, 921), (167, 165, 922), (165, 166, 924), (164, 167, 925), (162, 168, 927), (161, 169, 928), (159, 170, 930), (157, 171, 932), (155, 172, 934), (153, 173, 935), (151, 174, 937), (149, 175, 939), (146, 176, 942), (144, 177, 944), (142, 178, 946), (140, 179, 948), (139, 180, 949), (137, 181, 951), (136, 182, 952), (134, 183, 954), (133, 184, 955), (132, 185, 956), (131, 186, 957), (130, 187, 958), (129, 188, 959), (128, 189, 960), (127, 190, 960), (126, 191, 961), (126, 192, 961), (125, 193, 962), (124, 194, 963), (123, 195, 964), (122, 196, 965), (122, 197, 965), (121, 198, 966), (120, 199, 967), (119, 200, 968), (118, 201, 969), (117, 202, 970), (116, 203, 971), (114, 204, 973), (113, 205, 973), (112, 206, 974), (111, 207, 975), (109, 208, 977), (108, 209, 978), (107, 210, 979), (106, 211, 980), (105, 212, 981), (104, 213, 982), (103, 214, 983), (102, 215, 984), (101, 216, 985), (101, 217, 984), (100, 218, 985), (100, 219, 985), (99, 220, 986), (98, 221, 987), (98, 222, 987), (97, 223, 988), (97, 224, 987), (96, 225, 988), (96, 226, 988), (95, 227, 989), (95, 228, 989), (94, 229, 990), (94, 230, 990), (94, 231, 990), (93, 232, 990), (93, 233, 990), (92, 234, 991), (92, 235, 991), (92, 236, 991), (91, 237, 992), (91, 238, 991), (91, 239, 991), (91, 240, 991), (91, 241, 990), (90, 242, 991), (90, 243, 990), (90, 244, 990), (90, 245, 989), (90, 246, 989), (89, 247, 990), (89, 248, 989), (89, 249, 989), (89, 250, 988), (89, 251, 988), (88, 252, 988), (88, 253, 988), (88, 254, 987), (88, 255, 986), (88, 256, 986), (87, 257, 986), (87, 258, 986), (87, 259, 985), (87, 260, 984), (87, 261, 983), (86, 262, 983), (86, 263, 983), (86, 264, 982), (86, 265, 981), (85, 266, 981), (85, 267, 980), (85, 268, 980), (84, 269, 980), (84, 270, 979), (84, 271, 979), (84, 272, 978), (83, 273, 979), (83, 274, 978), (83, 275, 977), (82, 276, 978), (82, 277, 977), (82, 278, 977), (81, 279, 977), (81, 280, 977), (81, 281, 977), (80, 282, 977), (80, 283, 977), (80, 284, 976), (79, 285, 977), (79, 286, 976), (79, 287, 976), (78, 288, 976), (78, 289, 976), (78, 290, 975), (77, 291, 976), (77, 292, 975), (77, 293, 975), (76, 294, 975), (76, 295, 975), (76, 296, 974), (75, 297, 975), (75, 298, 974), (74, 299, 975), (74, 300, 974), (74, 301, 974), (73, 302, 974), (73, 303, 974), (72, 304, 974), (72, 305, 974), (71, 306, 974), (71, 307, 973), (71, 308, 972), (70, 309, 972), (70, 310, 971), (70, 311, 970), (70, 312, 968), (69, 313, 968), (69, 314, 966), (69, 315, 964), (69, 316, 962), (68, 317, 961), (68, 318, 959), (68, 319, 958), (68, 320, 956), (67, 321, 955), (67, 322, 954), (67, 323, 952), (67, 324, 951), (66, 325, 951), (66, 326, 950), (66, 327, 948), (66, 328, 947), (65, 329, 947), (65, 330, 946), (65, 331, 946), (65, 332, 945), (65, 333, 944), (65, 334, 942), (65, 335, 941), (65, 336, 940), (65, 337, 939), (65, 338, 938), (65, 339, 936), (64, 340, 936), (64, 341, 934), (64, 342, 932), (64, 343, 930), (64, 344, 928), (64, 345, 927), (64, 346, 925), (64, 347, 923), (64, 348, 922), (64, 349, 920), (64, 350, 919), (63, 351, 919), (63, 352, 918), (63, 353, 917), (63, 354, 916), (63, 355, 915), (63, 356, 914), (63, 357, 912), (63, 358, 911), (63, 359, 910), (63, 360, 909), (63, 361, 908), (63, 362, 906), (63, 363, 905), (63, 364, 904), (63, 365, 902), (63, 366, 901), (63, 367, 899), (63, 368, 898), (63, 369, 896), (62, 370, 895), (62, 371, 893), (62, 372, 891), (62, 373, 890), (62, 374, 888), (62, 375, 887), (62, 376, 886), (62, 377, 885), (62, 378, 884), (62, 379, 883), (63, 380, 880), (63, 381, 879), (63, 382, 878), (63, 383, 877), (63, 384, 876), (63, 385, 875), (63, 386, 874), (63, 387, 873), (63, 388, 872), (64, 389, 870), (64, 390, 869), (64, 391, 868), (64, 392, 867), (64, 393, 865), (64, 394, 864), (64, 395, 863), (65, 396, 861), (65, 397, 860), (65, 398, 859), (65, 399, 858), (65, 400, 857), (65, 401, 856), (65, 402, 854), (65, 403, 853), (65, 404, 851), (65, 405, 850), (65, 406, 848), (66, 407, 846), (66, 408, 844), (66, 409, 843), (66, 410, 842), (66, 411, 841), (66, 412, 840), (66, 413, 838), (66, 414, 837), (66, 415, 836), (66, 416, 835), (66, 417, 835), (66, 418, 834), (66, 419, 833), (67, 420, 831), (67, 421, 830), (67, 422, 829), (67, 423, 829), (67, 424, 828), (67, 425, 827), (67, 426, 826), (67, 427, 825), (67, 428, 824), (68, 429, 822), (68, 430, 820), (68, 431, 819), (68, 432, 818), (68, 433, 816), (68, 434, 815), (68, 435, 813), (68, 436, 811), (69, 437, 809), (69, 438, 807), (69, 439, 805), (69, 440, 804), (69, 441, 803), (69, 442, 802), (69, 443, 800), (70, 444, 798), (70, 445, 797), (70, 446, 796), (70, 447, 796), (71, 448, 794), (71, 449, 794), (72, 450, 792), (72, 451, 791), (73, 452, 790), (73, 453, 789), (74, 454, 788), (74, 455, 787), (75, 456, 786), (75, 457, 785), (76, 458, 784), (76, 459, 783), (77, 460, 782), (77, 461, 781), (77, 462, 781), (78, 463, 779), (78, 464, 779), (79, 465, 777), (79, 466, 777), (79, 467, 776), (80, 468, 775), (80, 469, 774), (80, 470, 774), (81, 471, 772), (81, 472, 771), (82, 473, 770), (82, 474, 769), (83, 475, 767), (83, 476, 766), (83, 477, 766), (84, 478, 764), (84, 479, 763), (85, 480, 761), (85, 481, 760), (85, 482, 759), (86, 483, 757), (86, 484, 755), (87, 485, 753), (87, 486, 752), (87, 487, 750), (88, 488, 748), (88, 489, 747), (88, 490, 746), (89, 491, 744), (89, 492, 743), (90, 493, 741), (90, 494, 741), (91, 495, 739), (91, 496, 738), (92, 497, 736), (93, 498, 735), (94, 499, 733), (94, 500, 733), (95, 501, 731), (96, 502, 729), (97, 503, 728), (98, 504, 726), (99, 505, 724), (99, 506, 724), (100, 507, 722), (101, 508, 721), (102, 509, 719), (104, 510, 717), (105, 511, 715), (106, 512, 714), (107, 513, 712), (108, 514, 711), (110, 515, 708), (111, 516, 707), (113, 517, 704), (114, 518, 703), (115, 519, 701), (117, 520, 698), (118, 521, 697), (119, 522, 695), (121, 523, 693), (122, 524, 691), (124, 525, 689), (125, 526, 687), (126, 527, 685), (128, 528, 683), (129, 529, 681), (131, 530, 678), (132, 531, 676), (134, 532, 673), (135, 533, 672), (137, 534, 669), (138, 535, 667), (140, 536, 664), (141, 537, 662), (143, 538, 659), (144, 539, 657), (146, 540, 654), (148, 541, 651), (149, 542, 649), (151, 543, 645), (153, 544, 642), (154, 545, 640), (156, 546, 638), (158, 547, 635), (159, 548, 633), (161, 549, 630), (162, 550, 628), (164, 551, 625), (166, 552, 623), (167, 553, 621), (169, 554, 618), (170, 555, 617), (171, 556, 615), (173, 557, 613), (174, 558, 611), (176, 559, 608), (177, 560, 607), (178, 561, 605), (180, 562, 603), (181, 563, 601), (183, 564, 599), (185, 565, 597), (186, 566, 595), (189, 567, 592), (192, 568, 589), (195, 569, 585), (198, 570, 582), (201, 571, 579), (204, 572, 575), (206, 573, 573), (209, 574, 569), (212, 575, 566), (215, 576, 563), (218, 577, 559), (221, 578, 556), (223, 579, 553), (226, 580, 550), (228, 581, 547), (230, 582, 545), (232, 583, 542), (234, 584, 540), (235, 585, 538), (237, 586, 536), (238, 587, 534), (240, 588, 531), (242, 589, 528), (243, 590, 526), (245, 591, 523), (247, 592, 520), (249, 593, 516), (251, 594, 513), (253, 595, 509), (256, 596, 505), (258, 597, 501), (261, 598, 497), (263, 599, 493), (267, 600, 488), (271, 601, 482), (274, 602, 478), (278, 603, 473), (281, 604, 468), (284, 605, 464), (287, 606, 460), (290, 607, 456), (292, 608, 453), (295, 609, 449), (298, 610, 445), (300, 611, 442), (303, 612, 438), (305, 613, 434), (308, 614, 430), (310, 615, 427), (312, 616, 423), (315, 617, 418), (317, 618, 415), (320, 619, 410), (322, 620, 406), (325, 621, 401), (327, 622, 396), (330, 623, 390), (333, 624, 384), (335, 625, 379), (338, 626, 374), (341, 627, 369), (345, 628, 362), (349, 629, 356), (353, 630, 350), (357, 631, 344), (360, 632, 340), (364, 633, 334), (368, 634, 328), (373, 635, 320), (378, 636, 313), (383, 637, 305), (389, 638, 295), (395, 639, 282), (401, 640, 270), (408, 641, 256), (416, 642, 240), (432, 643, 216), (448, 644, 193), (465, 645, 169), (480, 646, 148), (495, 647, 126), (511, 648, 104), (526, 649, 82), (565, 650, 9)], ['526,649,416,642,368,634,341,627,243,590,220,577,186,566,144,539,102,509,91,496,70,447,62,379,65,329,86,265,91,237,101,216,134,183,187,156,225,151,279,135,343,123,358,116,416,103,493,45,527,36,608,23,754,24,893,24,926,22,996,23,1032,27,1066,41,1082,52,1089,72,1088,172,1082,237,1045,305,1019,322,1002,338,950,373,889,429,865,446,851,473,830,493,810,528,786,554,772,586,740,612,683,638,607,649']), (917855882, 492601069, 445, 0, 438, 0, 116, 0.99195546, [(127, 1, 141), (94, 2, 206), (59, 3, 273), (338, 3, 59), (22, 4, 380), (19, 5, 386), (16, 6, 391), (15, 7, 393), (14, 8, 395), (14, 9, 396), (13, 10, 398), (12, 11, 399), (12, 12, 399), (11, 13, 401), (10, 14, 402), (11, 15, 402), (11, 16, 403), (12, 17, 403), (12, 18, 404), (12, 19, 405), (12, 20, 405), (12, 21, 405), (12, 22, 406), (12, 23, 406), (12, 24, 406), (12, 25, 407), (12, 26, 407), (12, 27, 407), (12, 28, 408), (12, 29, 408), (12, 30, 408), (12, 31, 408), (12, 32, 408), (12, 33, 408), (12, 34, 409), (12, 35, 409), (12, 36, 409), (12, 37, 409), (12, 38, 409), (12, 39, 409), (12, 40, 409), (12, 41, 409), (12, 42, 409), (12, 43, 410), (12, 44, 410), (12, 45, 409), (12, 46, 409), (12, 47, 409), (12, 48, 409), (12, 49, 409), (12, 50, 409), (12, 51, 409), (12, 52, 408), (12, 53, 408), (12, 54, 407), (12, 55, 406), (12, 56, 405), (12, 57, 404), (12, 58, 402), (11, 59, 402), (11, 60, 401), (11, 61, 401), (11, 62, 400), (11, 63, 399), (11, 64, 399), (11, 65, 398), (11, 66, 397), (11, 67, 396), (11, 68, 395), (11, 69, 395), (11, 70, 394), (11, 71, 394), (11, 72, 394), (11, 73, 393), (11, 74, 393), (11, 75, 393), (11, 76, 393), (11, 77, 393), (11, 78, 393), (11, 79, 392), (11, 80, 392), (10, 81, 393), (10, 82, 394), (10, 83, 394), (9, 84, 396), (9, 85, 263), (284, 85, 121), (9, 86, 75), (97, 86, 29), (141, 86, 120), (292, 86, 112), (9, 87, 71), (151, 87, 105), (294, 87, 110), (8, 88, 67), (160, 88, 92), (295, 88, 108), (8, 89, 63), (175, 89, 74), (296, 89, 107), (7, 90, 61), (204, 90, 41), (297, 90, 105), (7, 91, 57), (298, 91, 104), (7, 92, 53), (299, 92, 103), (6, 93, 50), (300, 93, 101), (7, 94, 46), (303, 94, 96), (7, 95, 44), (305, 95, 93), (7, 96, 42), (308, 96, 88), (7, 97, 40), (310, 97, 85), (7, 98, 38), (312, 98, 82), (8, 99, 34), (314, 99, 79), (8, 100, 32), (316, 100, 75), (8, 101, 29), (319, 101, 71), (13, 102, 19), (324, 102, 63), (20, 103, 6), (331, 103, 51), (337, 104, 37), (344, 105, 22), (351, 106, 2)], ['344,105,319,101,307,95,300,93,283,84,261,85,244,90,204,90,203,89,175,89,160,88,140,85,125,86,97,86,84,85,67,90,56,92,36,101,26,102,8,101,6,93,11,80,11,59,12,58,12,17,10,14,16,6,22,4,58,4,59,3,93,3,94,2,126,2,127,1,267,1,268,2,331,3,396,3,406,6,416,19,420,34,420,51,411,60,405,68,402,81,404,85,401,92,386,102,374,103,365,105']), (917855882, 492601069, 445, 390, 550, 0, 54, 0.9391579, [(414, 0, 7), (441, 0, 60), (508, 0, 28), (402, 1, 142), (401, 2, 146), (402, 3, 145), (404, 4, 143), (406, 5, 140), (408, 6, 137), (410, 7, 134), (411, 8, 132), (412, 9, 130), (413, 10, 127), (414, 11, 125), (415, 12, 123), (415, 13, 122), (416, 14, 120), (417, 15, 117), (417, 16, 116), (418, 17, 114), (418, 18, 113), (418, 19, 111), (418, 20, 109), (419, 21, 107), (419, 22, 105), (419, 23, 103), (419, 24, 102), (419, 25, 100), (420, 26, 97), (420, 27, 95), (420, 28, 94), (421, 29, 91), (421, 30, 90), (422, 31, 88), (422, 32, 88), (422, 33, 87), (423, 34, 84), (423, 35, 82), (423, 36, 81), (424, 37, 79), (424, 38, 77), (424, 39, 75), (424, 40, 73), (424, 41, 71), (425, 42, 67), (425, 43, 66), (426, 44, 62), (426, 45, 6), (433, 45, 52), (443, 46, 30), (450, 47, 1)], ['450,47,449,46,443,46,442,45,426,45,424,41,424,37,423,36,422,31,419,25,419,21,418,20,418,17,417,15,409,6,402,3,402,1,413,1,414,0,420,0,421,1,440,1,441,0,500,0,501,1,507,1,508,0,535,0,536,1,543,1,546,2,546,4,542,8,530,18,527,19,525,21,522,22,520,24,512,28,508,33,505,34,502,37,494,41,492,41,490,43,488,43,484,45,473,45,472,46,451,46'])], 'temp/1742247330_763596_917855882_da0fa7b7e6b5b551fe26c0ba8713276d.jpg']} ############################### TEST POLYGON ################################ Inside batchDatouExec : verbose : False # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! List Step Type Loaded in datou : mask_detect list_input_json : [] origin BFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 time to download the photos : 0.1392378807067871 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : False number of steps : 1 step1:mask_detect Mon Mar 17 22:35:57 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3000 l 3632 free memory gpu now : 568 wait 20 seconds l 3637 free memory gpu now : 568 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-03-17 22:36:20.598433: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-17 22:36:20.627175: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-17 22:36:20.629305: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fc640000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-17 22:36:20.629357: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-17 22:36:20.632944: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-17 22:36:20.772845: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xcba1170 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-17 22:36:20.772893: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-17 22:36:20.773769: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-17 22:36:20.774159: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-17 22:36:20.777045: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-17 22:36:20.779590: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-17 22:36:20.780072: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-17 22:36:20.783128: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-17 22:36:20.784517: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-17 22:36:20.788511: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-17 22:36:20.789385: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-17 22:36:20.789455: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-17 22:36:20.789927: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-17 22:36:20.789942: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-17 22:36:20.789951: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-17 22:36:20.790744: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 191 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-03-17 22:36:20.866619: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-17 22:36:20.866709: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-17 22:36:20.866731: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-17 22:36:20.866751: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-17 22:36:20.866771: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-17 22:36:20.866790: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-17 22:36:20.866808: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-17 22:36:20.866828: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-17 22:36:20.867537: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-17 22:36:20.868375: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-17 22:36:20.868417: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-17 22:36:20.868441: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-17 22:36:20.868464: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-17 22:36:20.868502: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-17 22:36:20.868525: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-17 22:36:20.868548: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-17 22:36:20.868571: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-17 22:36:20.869371: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-17 22:36:20.869400: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-17 22:36:20.869411: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-17 22:36:20.869420: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-17 22:36:20.870269: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 191 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) 2025-03-17 22:36:32.562456: W tensorflow/core/common_runtime/bfc_allocator.cc:434] Allocator (GPU_0_bfc) ran out of memory trying to allocate 9.00MiB (rounded to 9437184) Current allocation summary follows. 2025-03-17 22:36:32.562557: I tensorflow/core/common_runtime/bfc_allocator.cc:934] BFCAllocator dump for GPU_0_bfc 2025-03-17 22:36:32.562578: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (256): Total Chunks: 65, Chunks in use: 65. 16.2KiB allocated for chunks. 16.2KiB in use in bin. 8.9KiB client-requested in use in bin. 2025-03-17 22:36:32.562595: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (512): Total Chunks: 40, Chunks in use: 40. 20.0KiB allocated for chunks. 20.0KiB in use in bin. 20.0KiB client-requested in use in bin. 2025-03-17 22:36:32.562611: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1024): Total Chunks: 251, Chunks in use: 251. 251.2KiB allocated for chunks. 251.2KiB in use in bin. 251.0KiB client-requested in use in bin. 2025-03-17 22:36:32.562635: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2048): Total Chunks: 40, Chunks in use: 40. 80.0KiB allocated for chunks. 80.0KiB in use in bin. 80.0KiB client-requested in use in bin. 2025-03-17 22:36:32.562650: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4096): Total Chunks: 120, Chunks in use: 120. 487.5KiB allocated for chunks. 487.5KiB in use in bin. 480.0KiB client-requested in use in bin. 2025-03-17 22:36:32.562665: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8192): Total Chunks: 10, Chunks in use: 10. 80.0KiB allocated for chunks. 80.0KiB in use in bin. 80.0KiB client-requested in use in bin. 2025-03-17 22:36:32.562680: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16384): Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin. 2025-03-17 22:36:32.562695: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (32768): Total Chunks: 1, Chunks in use: 1. 36.8KiB allocated for chunks. 36.8KiB in use in bin. 36.8KiB client-requested in use in bin. 2025-03-17 22:36:32.562710: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (65536): Total Chunks: 6, Chunks in use: 6. 384.0KiB allocated for chunks. 384.0KiB in use in bin. 384.0KiB client-requested in use in bin. 2025-03-17 22:36:32.562725: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (131072): Total Chunks: 4, Chunks in use: 4. 560.0KiB allocated for chunks. 560.0KiB in use in bin. 560.0KiB client-requested in use in bin. 2025-03-17 22:36:32.562740: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (262144): Total Chunks: 8, Chunks in use: 7. 2.35MiB allocated for chunks. 1.91MiB in use in bin. 1.75MiB client-requested in use in bin. 2025-03-17 22:36:32.562777: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (524288): Total Chunks: 7, Chunks in use: 6. 4.39MiB allocated for chunks. 3.44MiB in use in bin. 3.25MiB client-requested in use in bin. 2025-03-17 22:36:32.562794: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (1048576): Total Chunks: 48, Chunks in use: 45. 54.12MiB allocated for chunks. 50.62MiB in use in bin. 45.00MiB client-requested in use in bin. 2025-03-17 22:36:32.562809: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (2097152): Total Chunks: 28, Chunks in use: 25. 64.25MiB allocated for chunks. 55.75MiB in use in bin. 55.75MiB client-requested in use in bin. 2025-03-17 22:36:32.562824: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (4194304): Total Chunks: 4, Chunks in use: 2. 18.00MiB allocated for chunks. 8.00MiB in use in bin. 8.00MiB client-requested in use in bin. 2025-03-17 22:36:32.562838: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (8388608): Total Chunks: 5, Chunks in use: 4. 46.88MiB allocated for chunks. 38.88MiB in use in bin. 35.00MiB client-requested in use in bin. 2025-03-17 22:36:32.562852: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-03-17 22:36:32.562866: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-03-17 22:36:32.562879: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-03-17 22:36:32.562892: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-03-17 22:36:32.562929: I tensorflow/core/common_runtime/bfc_allocator.cc:941] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2025-03-17 22:36:32.562948: I tensorflow/core/common_runtime/bfc_allocator.cc:957] Bin for 9.00MiB was 8.00MiB, Chunk State: 2025-03-17 22:36:32.562977: I tensorflow/core/common_runtime/bfc_allocator.cc:963] Size: 8.00MiB | Requested Size: 8.00MiB | in_use: 0 | bin_num: 15, prev: Size: 9.00MiB | Requested Size: 9.00MiB | in_use: 1 | bin_num: -1, next: Size: 8.00MiB | Requested Size: 8.00MiB | in_use: 1 | bin_num: -1 2025-03-17 22:36:32.562990: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 68026368 2025-03-17 22:36:32.563005: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fc45e000000 of size 4194304 next 632 2025-03-17 22:36:32.563018: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc45e400000 of size 4194304 next 617 2025-03-17 22:36:32.563030: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc45e800000 of size 4194304 next 616 2025-03-17 22:36:32.563041: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fc45ec00000 of size 6291456 next 608 2025-03-17 22:36:32.563054: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc45f200000 of size 9437184 next 607 2025-03-17 22:36:32.563065: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fc45fb00000 of size 8388608 next 625 2025-03-17 22:36:32.563077: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc460300000 of size 8388608 next 624 2025-03-17 22:36:32.563088: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc460b00000 of size 9437184 next 637 2025-03-17 22:36:32.563100: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc461400000 of size 13500416 next 18446744073709551615 2025-03-17 22:36:32.563122: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 67108864 2025-03-17 22:36:32.563134: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc464000000 of size 2359296 next 370 2025-03-17 22:36:32.563146: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc464240000 of size 1048576 next 394 2025-03-17 22:36:32.563157: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fc464340000 of size 1310720 next 388 2025-03-17 22:36:32.563169: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc464480000 of size 2359296 next 387 2025-03-17 22:36:32.563180: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc4646c0000 of size 1048576 next 412 2025-03-17 22:36:32.563192: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc4647c0000 of size 1310720 next 406 2025-03-17 22:36:32.563204: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc464900000 of size 2359296 next 405 2025-03-17 22:36:32.563215: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc464b40000 of size 1048576 next 430 2025-03-17 22:36:32.563227: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc464c40000 of size 1310720 next 424 2025-03-17 22:36:32.563238: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc464d80000 of size 2359296 next 423 2025-03-17 22:36:32.563250: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc464fc0000 of size 1048576 next 447 2025-03-17 22:36:32.563261: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc4650c0000 of size 1310720 next 441 2025-03-17 22:36:32.563272: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc465200000 of size 2359296 next 440 2025-03-17 22:36:32.563284: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc465440000 of size 1048576 next 465 2025-03-17 22:36:32.563295: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc465540000 of size 1310720 next 459 2025-03-17 22:36:32.563307: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc465680000 of size 2359296 next 458 2025-03-17 22:36:32.563318: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc4658c0000 of size 1048576 next 483 2025-03-17 22:36:32.563329: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc4659c0000 of size 1310720 next 477 2025-03-17 22:36:32.563341: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc465b00000 of size 2359296 next 476 2025-03-17 22:36:32.563352: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc465d40000 of size 1048576 next 501 2025-03-17 22:36:32.563364: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc465e40000 of size 1310720 next 495 2025-03-17 22:36:32.563375: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc465f80000 of size 2359296 next 494 2025-03-17 22:36:32.563387: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc4661c0000 of size 1048576 next 519 2025-03-17 22:36:32.563398: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc4662c0000 of size 1310720 next 513 2025-03-17 22:36:32.563409: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc466400000 of size 2359296 next 512 2025-03-17 22:36:32.563421: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc466640000 of size 1048576 next 537 2025-03-17 22:36:32.563432: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc466740000 of size 1310720 next 531 2025-03-17 22:36:32.563444: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc466880000 of size 2359296 next 530 2025-03-17 22:36:32.563455: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc466ac0000 of size 1048576 next 555 2025-03-17 22:36:32.563466: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc466bc0000 of size 1310720 next 549 2025-03-17 22:36:32.563478: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc466d00000 of size 2359296 next 548 2025-03-17 22:36:32.563498: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc466f40000 of size 1048576 next 573 2025-03-17 22:36:32.563509: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc467040000 of size 1310720 next 567 2025-03-17 22:36:32.563521: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc467180000 of size 2359296 next 566 2025-03-17 22:36:32.563532: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc4673c0000 of size 1048576 next 591 2025-03-17 22:36:32.563544: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fc4674c0000 of size 1310720 next 585 2025-03-17 22:36:32.563555: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc467600000 of size 2359296 next 584 2025-03-17 22:36:32.563566: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fc467840000 of size 2097152 next 597 2025-03-17 22:36:32.563579: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc467a40000 of size 2097152 next 596 2025-03-17 22:36:32.563590: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fc467c40000 of size 3932160 next 18446744073709551615 2025-03-17 22:36:32.563602: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 33554432 2025-03-17 22:36:32.563614: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc468000000 of size 2359296 next 245 2025-03-17 22:36:32.563625: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc468240000 of size 1048576 next 269 2025-03-17 22:36:32.563636: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc468340000 of size 1310720 next 263 2025-03-17 22:36:32.563648: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc468480000 of size 2359296 next 262 2025-03-17 22:36:32.563659: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc4686c0000 of size 1048576 next 287 2025-03-17 22:36:32.563671: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc4687c0000 of size 1310720 next 281 2025-03-17 22:36:32.563682: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc468900000 of size 2359296 next 280 2025-03-17 22:36:32.563694: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc468b40000 of size 1048576 next 304 2025-03-17 22:36:32.563705: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc468c40000 of size 1310720 next 298 2025-03-17 22:36:32.563716: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc468d80000 of size 2359296 next 297 2025-03-17 22:36:32.563728: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc468fc0000 of size 1048576 next 322 2025-03-17 22:36:32.563739: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc4690c0000 of size 1310720 next 316 2025-03-17 22:36:32.563751: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc469200000 of size 2359296 next 315 2025-03-17 22:36:32.563762: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc469440000 of size 1048576 next 340 2025-03-17 22:36:32.563773: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc469540000 of size 1310720 next 334 2025-03-17 22:36:32.563785: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc469680000 of size 2359296 next 333 2025-03-17 22:36:32.563796: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc4698c0000 of size 1048576 next 358 2025-03-17 22:36:32.563807: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc4699c0000 of size 1310720 next 352 2025-03-17 22:36:32.563819: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc469b00000 of size 2359296 next 351 2025-03-17 22:36:32.563830: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc469d40000 of size 1048576 next 376 2025-03-17 22:36:32.563842: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc469e40000 of size 1835008 next 18446744073709551615 2025-03-17 22:36:32.563861: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 16777216 2025-03-17 22:36:32.563874: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46a000000 of size 2359296 next 181 2025-03-17 22:36:32.563885: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46a240000 of size 2097152 next 196 2025-03-17 22:36:32.563897: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46a440000 of size 1048576 next 216 2025-03-17 22:36:32.563908: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46a540000 of size 1310720 next 209 2025-03-17 22:36:32.563920: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46a680000 of size 2359296 next 208 2025-03-17 22:36:32.563931: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46a8c0000 of size 1048576 next 234 2025-03-17 22:36:32.563943: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46a9c0000 of size 1310720 next 228 2025-03-17 22:36:32.563954: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46ab00000 of size 2359296 next 227 2025-03-17 22:36:32.563965: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fc46ad40000 of size 2883584 next 18446744073709551615 2025-03-17 22:36:32.563977: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 2097152 2025-03-17 22:36:32.563988: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c600000 of size 147456 next 55 2025-03-17 22:36:32.564000: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c624000 of size 65536 next 78 2025-03-17 22:36:32.564012: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c634000 of size 4096 next 191 2025-03-17 22:36:32.564023: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c635000 of size 4096 next 192 2025-03-17 22:36:32.564035: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c636000 of size 4096 next 193 2025-03-17 22:36:32.564047: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c637000 of size 256 next 194 2025-03-17 22:36:32.564058: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c637100 of size 256 next 195 2025-03-17 22:36:32.564070: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c637200 of size 4096 next 197 2025-03-17 22:36:32.564081: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c638200 of size 4096 next 198 2025-03-17 22:36:32.564092: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c639200 of size 4096 next 199 2025-03-17 22:36:32.564104: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c63a200 of size 4096 next 200 2025-03-17 22:36:32.564115: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c63b200 of size 4096 next 201 2025-03-17 22:36:32.564127: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c63c200 of size 1024 next 202 2025-03-17 22:36:32.564138: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c63c600 of size 1024 next 204 2025-03-17 22:36:32.564150: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c63ca00 of size 1024 next 205 2025-03-17 22:36:32.564161: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c63ce00 of size 1024 next 206 2025-03-17 22:36:32.564173: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c63d200 of size 1024 next 207 2025-03-17 22:36:32.564184: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c63d600 of size 1024 next 210 2025-03-17 22:36:32.564196: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c63da00 of size 1024 next 211 2025-03-17 22:36:32.564207: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c63de00 of size 1024 next 212 2025-03-17 22:36:32.564218: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c63e200 of size 1024 next 213 2025-03-17 22:36:32.564230: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c63e600 of size 1024 next 214 2025-03-17 22:36:32.564249: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c63ea00 of size 4096 next 215 2025-03-17 22:36:32.564261: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c63fa00 of size 4096 next 217 2025-03-17 22:36:32.564272: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c640a00 of size 4096 next 218 2025-03-17 22:36:32.564284: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c641a00 of size 4096 next 219 2025-03-17 22:36:32.564295: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c642a00 of size 4096 next 220 2025-03-17 22:36:32.564307: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c643a00 of size 1024 next 221 2025-03-17 22:36:32.564318: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c643e00 of size 1024 next 223 2025-03-17 22:36:32.564330: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c644200 of size 1024 next 224 2025-03-17 22:36:32.564341: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c644600 of size 1024 next 225 2025-03-17 22:36:32.564352: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c644a00 of size 1024 next 226 2025-03-17 22:36:32.564364: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c644e00 of size 1024 next 229 2025-03-17 22:36:32.564375: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c645200 of size 1024 next 230 2025-03-17 22:36:32.564387: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c645600 of size 1024 next 231 2025-03-17 22:36:32.564398: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c645a00 of size 1024 next 232 2025-03-17 22:36:32.564409: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c645e00 of size 1024 next 233 2025-03-17 22:36:32.564421: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c646200 of size 7680 next 72 2025-03-17 22:36:32.564433: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c648000 of size 147456 next 71 2025-03-17 22:36:32.564444: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c66c000 of size 131072 next 87 2025-03-17 22:36:32.564456: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c68c000 of size 4096 next 289 2025-03-17 22:36:32.564467: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c68d000 of size 4096 next 290 2025-03-17 22:36:32.564479: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c68e000 of size 4096 next 291 2025-03-17 22:36:32.564490: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c68f000 of size 1024 next 292 2025-03-17 22:36:32.564501: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c68f400 of size 1024 next 293 2025-03-17 22:36:32.564513: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c68f800 of size 1024 next 294 2025-03-17 22:36:32.564524: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c68fc00 of size 1024 next 295 2025-03-17 22:36:32.564536: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c690000 of size 1024 next 296 2025-03-17 22:36:32.564547: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c690400 of size 1024 next 299 2025-03-17 22:36:32.564558: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c690800 of size 1024 next 300 2025-03-17 22:36:32.564570: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c690c00 of size 1024 next 301 2025-03-17 22:36:32.564581: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c691000 of size 1024 next 302 2025-03-17 22:36:32.564593: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c691400 of size 1024 next 303 2025-03-17 22:36:32.564604: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c691800 of size 4096 next 305 2025-03-17 22:36:32.564623: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c692800 of size 4096 next 306 2025-03-17 22:36:32.564635: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c693800 of size 4096 next 307 2025-03-17 22:36:32.564647: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c694800 of size 4096 next 308 2025-03-17 22:36:32.564658: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c695800 of size 4096 next 309 2025-03-17 22:36:32.564670: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c696800 of size 1024 next 310 2025-03-17 22:36:32.564681: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c696c00 of size 1024 next 311 2025-03-17 22:36:32.564692: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c697000 of size 1024 next 312 2025-03-17 22:36:32.564704: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c697400 of size 1024 next 313 2025-03-17 22:36:32.564715: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c697800 of size 1024 next 314 2025-03-17 22:36:32.564727: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c697c00 of size 1024 next 317 2025-03-17 22:36:32.564738: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c698000 of size 1024 next 318 2025-03-17 22:36:32.564750: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c698400 of size 1024 next 319 2025-03-17 22:36:32.564761: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c698800 of size 1024 next 320 2025-03-17 22:36:32.564772: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c698c00 of size 1024 next 321 2025-03-17 22:36:32.564784: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c699000 of size 4096 next 323 2025-03-17 22:36:32.564795: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c69a000 of size 4096 next 324 2025-03-17 22:36:32.564807: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c69b000 of size 4096 next 325 2025-03-17 22:36:32.564818: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c69c000 of size 4096 next 326 2025-03-17 22:36:32.564829: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c69d000 of size 4096 next 327 2025-03-17 22:36:32.564841: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c69e000 of size 1024 next 328 2025-03-17 22:36:32.564852: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c69e400 of size 1024 next 329 2025-03-17 22:36:32.564864: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c69e800 of size 1024 next 330 2025-03-17 22:36:32.564875: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c69ec00 of size 1024 next 331 2025-03-17 22:36:32.564886: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c69f000 of size 1024 next 332 2025-03-17 22:36:32.564898: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c69f400 of size 1024 next 335 2025-03-17 22:36:32.564909: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c69f800 of size 1024 next 336 2025-03-17 22:36:32.564920: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c69fc00 of size 1024 next 337 2025-03-17 22:36:32.564932: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6a0000 of size 1024 next 338 2025-03-17 22:36:32.564943: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6a0400 of size 1024 next 339 2025-03-17 22:36:32.564955: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6a0800 of size 4096 next 341 2025-03-17 22:36:32.564966: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6a1800 of size 4096 next 342 2025-03-17 22:36:32.564977: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6a2800 of size 4096 next 343 2025-03-17 22:36:32.564997: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6a3800 of size 4096 next 344 2025-03-17 22:36:32.565009: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6a4800 of size 4096 next 345 2025-03-17 22:36:32.565020: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6a5800 of size 1024 next 346 2025-03-17 22:36:32.565032: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6a5c00 of size 1024 next 347 2025-03-17 22:36:32.565043: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6a6000 of size 1024 next 348 2025-03-17 22:36:32.565055: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6a6400 of size 1024 next 349 2025-03-17 22:36:32.565066: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6a6800 of size 1024 next 350 2025-03-17 22:36:32.565078: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6a6c00 of size 1024 next 353 2025-03-17 22:36:32.565089: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6a7000 of size 1024 next 354 2025-03-17 22:36:32.565101: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6a7400 of size 1024 next 355 2025-03-17 22:36:32.565112: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6a7800 of size 1024 next 356 2025-03-17 22:36:32.565123: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6a7c00 of size 1024 next 357 2025-03-17 22:36:32.565135: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6a8000 of size 4096 next 359 2025-03-17 22:36:32.565146: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6a9000 of size 4096 next 360 2025-03-17 22:36:32.565157: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6aa000 of size 4096 next 361 2025-03-17 22:36:32.565169: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6ab000 of size 4096 next 362 2025-03-17 22:36:32.565180: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6ac000 of size 4096 next 363 2025-03-17 22:36:32.565192: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6ad000 of size 1024 next 364 2025-03-17 22:36:32.565203: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6ad400 of size 1024 next 365 2025-03-17 22:36:32.565214: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6ad800 of size 1024 next 366 2025-03-17 22:36:32.565226: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6adc00 of size 1024 next 367 2025-03-17 22:36:32.565237: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6ae000 of size 1024 next 368 2025-03-17 22:36:32.565249: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6ae400 of size 1024 next 371 2025-03-17 22:36:32.565260: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6ae800 of size 1024 next 372 2025-03-17 22:36:32.565271: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6aec00 of size 1024 next 373 2025-03-17 22:36:32.565283: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6af000 of size 1024 next 374 2025-03-17 22:36:32.565294: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6af400 of size 1024 next 375 2025-03-17 22:36:32.565306: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6af800 of size 4096 next 377 2025-03-17 22:36:32.565317: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6b0800 of size 4096 next 378 2025-03-17 22:36:32.565328: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6b1800 of size 4096 next 379 2025-03-17 22:36:32.565340: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6b2800 of size 4096 next 380 2025-03-17 22:36:32.565351: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6b3800 of size 4096 next 381 2025-03-17 22:36:32.565370: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6b4800 of size 1024 next 382 2025-03-17 22:36:32.565382: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6b4c00 of size 1024 next 383 2025-03-17 22:36:32.565394: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6b5000 of size 1024 next 384 2025-03-17 22:36:32.565405: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6b5400 of size 1024 next 385 2025-03-17 22:36:32.565417: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6b5800 of size 1024 next 386 2025-03-17 22:36:32.565428: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6b5c00 of size 1024 next 389 2025-03-17 22:36:32.565439: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6b6000 of size 1024 next 390 2025-03-17 22:36:32.565451: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6b6400 of size 1024 next 391 2025-03-17 22:36:32.565462: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6b6800 of size 1024 next 392 2025-03-17 22:36:32.565474: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6b6c00 of size 1024 next 393 2025-03-17 22:36:32.565485: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6b7000 of size 4096 next 395 2025-03-17 22:36:32.565496: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6b8000 of size 4096 next 396 2025-03-17 22:36:32.565508: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6b9000 of size 4096 next 397 2025-03-17 22:36:32.565519: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6ba000 of size 4096 next 398 2025-03-17 22:36:32.565531: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6bb000 of size 4096 next 399 2025-03-17 22:36:32.565542: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6bc000 of size 1024 next 400 2025-03-17 22:36:32.565554: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6bc400 of size 1024 next 401 2025-03-17 22:36:32.565565: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6bc800 of size 1024 next 402 2025-03-17 22:36:32.565576: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6bcc00 of size 1024 next 403 2025-03-17 22:36:32.565588: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6bd000 of size 1024 next 404 2025-03-17 22:36:32.565599: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6bd400 of size 1024 next 407 2025-03-17 22:36:32.565610: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6bd800 of size 1024 next 408 2025-03-17 22:36:32.565622: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6bdc00 of size 1024 next 409 2025-03-17 22:36:32.565633: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6be000 of size 1024 next 410 2025-03-17 22:36:32.565645: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6be400 of size 1024 next 411 2025-03-17 22:36:32.565656: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6be800 of size 4096 next 413 2025-03-17 22:36:32.565667: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6bf800 of size 4096 next 414 2025-03-17 22:36:32.565679: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6c0800 of size 4096 next 415 2025-03-17 22:36:32.565690: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6c1800 of size 4096 next 416 2025-03-17 22:36:32.565702: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6c2800 of size 4096 next 417 2025-03-17 22:36:32.565713: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6c3800 of size 1024 next 418 2025-03-17 22:36:32.565725: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6c3c00 of size 1024 next 419 2025-03-17 22:36:32.565736: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6c4000 of size 1024 next 420 2025-03-17 22:36:32.565756: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6c4400 of size 1024 next 421 2025-03-17 22:36:32.565767: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6c4800 of size 1024 next 422 2025-03-17 22:36:32.565779: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6c4c00 of size 1024 next 425 2025-03-17 22:36:32.565790: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6c5000 of size 1024 next 426 2025-03-17 22:36:32.565802: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6c5400 of size 1024 next 427 2025-03-17 22:36:32.565813: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6c5800 of size 1024 next 428 2025-03-17 22:36:32.565824: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6c5c00 of size 1024 next 429 2025-03-17 22:36:32.565836: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6c6000 of size 4096 next 431 2025-03-17 22:36:32.565847: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6c7000 of size 4096 next 432 2025-03-17 22:36:32.565859: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6c8000 of size 4096 next 433 2025-03-17 22:36:32.565870: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6c9000 of size 4096 next 434 2025-03-17 22:36:32.565881: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6ca000 of size 4096 next 435 2025-03-17 22:36:32.565893: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6cb000 of size 1024 next 436 2025-03-17 22:36:32.565904: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6cb400 of size 1024 next 437 2025-03-17 22:36:32.565916: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6cb800 of size 1024 next 438 2025-03-17 22:36:32.565927: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6cbc00 of size 1024 next 103 2025-03-17 22:36:32.565939: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c6cc000 of size 262144 next 102 2025-03-17 22:36:32.565951: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fc46c70c000 of size 999424 next 18446744073709551615 2025-03-17 22:36:32.565962: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 4194304 2025-03-17 22:36:32.565974: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c800000 of size 589824 next 96 2025-03-17 22:36:32.565986: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c890000 of size 262144 next 117 2025-03-17 22:36:32.565997: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8d0000 of size 1024 next 439 2025-03-17 22:36:32.566008: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8d0400 of size 1024 next 442 2025-03-17 22:36:32.566020: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8d0800 of size 1024 next 443 2025-03-17 22:36:32.566031: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8d0c00 of size 1024 next 444 2025-03-17 22:36:32.566043: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8d1000 of size 1024 next 445 2025-03-17 22:36:32.566054: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8d1400 of size 1024 next 446 2025-03-17 22:36:32.566065: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8d1800 of size 4096 next 448 2025-03-17 22:36:32.566077: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8d2800 of size 4096 next 449 2025-03-17 22:36:32.566088: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8d3800 of size 4096 next 450 2025-03-17 22:36:32.566099: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8d4800 of size 4096 next 451 2025-03-17 22:36:32.566111: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8d5800 of size 4096 next 452 2025-03-17 22:36:32.566130: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8d6800 of size 1024 next 453 2025-03-17 22:36:32.566142: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8d6c00 of size 1024 next 454 2025-03-17 22:36:32.566154: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8d7000 of size 1024 next 455 2025-03-17 22:36:32.566165: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8d7400 of size 1024 next 456 2025-03-17 22:36:32.566176: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8d7800 of size 1024 next 457 2025-03-17 22:36:32.566188: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8d7c00 of size 1024 next 460 2025-03-17 22:36:32.566199: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8d8000 of size 1024 next 461 2025-03-17 22:36:32.566211: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8d8400 of size 1024 next 462 2025-03-17 22:36:32.566222: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8d8800 of size 1024 next 463 2025-03-17 22:36:32.566233: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8d8c00 of size 1024 next 464 2025-03-17 22:36:32.566245: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8d9000 of size 4096 next 466 2025-03-17 22:36:32.566256: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8da000 of size 4096 next 467 2025-03-17 22:36:32.566268: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8db000 of size 4096 next 468 2025-03-17 22:36:32.566279: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8dc000 of size 4096 next 469 2025-03-17 22:36:32.566290: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8dd000 of size 4096 next 470 2025-03-17 22:36:32.566302: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8de000 of size 1024 next 471 2025-03-17 22:36:32.566313: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8de400 of size 1024 next 472 2025-03-17 22:36:32.566325: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8de800 of size 1024 next 473 2025-03-17 22:36:32.566336: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8dec00 of size 1024 next 474 2025-03-17 22:36:32.566348: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8df000 of size 1024 next 475 2025-03-17 22:36:32.566359: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8df400 of size 1024 next 478 2025-03-17 22:36:32.566370: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8df800 of size 1024 next 479 2025-03-17 22:36:32.566382: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8dfc00 of size 1024 next 480 2025-03-17 22:36:32.566393: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8e0000 of size 1024 next 481 2025-03-17 22:36:32.566404: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8e0400 of size 1024 next 482 2025-03-17 22:36:32.566416: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8e0800 of size 4096 next 484 2025-03-17 22:36:32.566427: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8e1800 of size 4096 next 485 2025-03-17 22:36:32.566439: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8e2800 of size 4096 next 486 2025-03-17 22:36:32.566450: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8e3800 of size 4096 next 487 2025-03-17 22:36:32.566461: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8e4800 of size 4096 next 488 2025-03-17 22:36:32.566473: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8e5800 of size 1024 next 489 2025-03-17 22:36:32.566484: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8e5c00 of size 1024 next 490 2025-03-17 22:36:32.566503: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8e6000 of size 1024 next 491 2025-03-17 22:36:32.566515: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8e6400 of size 1024 next 492 2025-03-17 22:36:32.566527: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8e6800 of size 1024 next 493 2025-03-17 22:36:32.566538: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8e6c00 of size 1024 next 496 2025-03-17 22:36:32.566549: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8e7000 of size 1024 next 497 2025-03-17 22:36:32.566561: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8e7400 of size 1024 next 498 2025-03-17 22:36:32.566572: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8e7800 of size 1024 next 499 2025-03-17 22:36:32.566584: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8e7c00 of size 1024 next 500 2025-03-17 22:36:32.566595: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8e8000 of size 4096 next 502 2025-03-17 22:36:32.566606: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8e9000 of size 4096 next 503 2025-03-17 22:36:32.566618: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8ea000 of size 4096 next 504 2025-03-17 22:36:32.566629: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8eb000 of size 4096 next 505 2025-03-17 22:36:32.566641: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8ec000 of size 4096 next 506 2025-03-17 22:36:32.566652: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8ed000 of size 1024 next 507 2025-03-17 22:36:32.566664: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8ed400 of size 1024 next 508 2025-03-17 22:36:32.566675: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8ed800 of size 1024 next 509 2025-03-17 22:36:32.566686: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8edc00 of size 1024 next 510 2025-03-17 22:36:32.566698: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8ee000 of size 1024 next 511 2025-03-17 22:36:32.566709: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8ee400 of size 1024 next 514 2025-03-17 22:36:32.566721: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8ee800 of size 1024 next 515 2025-03-17 22:36:32.566732: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8eec00 of size 1024 next 516 2025-03-17 22:36:32.566743: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8ef000 of size 1024 next 517 2025-03-17 22:36:32.566755: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8ef400 of size 1024 next 518 2025-03-17 22:36:32.566766: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8ef800 of size 4096 next 520 2025-03-17 22:36:32.566778: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8f0800 of size 4096 next 521 2025-03-17 22:36:32.566789: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8f1800 of size 4096 next 522 2025-03-17 22:36:32.566800: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8f2800 of size 4096 next 523 2025-03-17 22:36:32.566812: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8f3800 of size 4096 next 524 2025-03-17 22:36:32.566823: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8f4800 of size 1024 next 525 2025-03-17 22:36:32.566835: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8f4c00 of size 1024 next 526 2025-03-17 22:36:32.566846: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8f5000 of size 1024 next 527 2025-03-17 22:36:32.566857: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8f5400 of size 1024 next 528 2025-03-17 22:36:32.566877: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8f5800 of size 1024 next 529 2025-03-17 22:36:32.566889: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8f5c00 of size 1024 next 532 2025-03-17 22:36:32.566913: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8f6000 of size 1024 next 533 2025-03-17 22:36:32.566925: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8f6400 of size 1024 next 534 2025-03-17 22:36:32.566936: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8f6800 of size 1024 next 535 2025-03-17 22:36:32.566948: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8f6c00 of size 1024 next 536 2025-03-17 22:36:32.566959: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8f7000 of size 4096 next 538 2025-03-17 22:36:32.566971: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8f8000 of size 4096 next 539 2025-03-17 22:36:32.566982: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8f9000 of size 4096 next 540 2025-03-17 22:36:32.566994: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8fa000 of size 4096 next 541 2025-03-17 22:36:32.567005: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8fb000 of size 4096 next 542 2025-03-17 22:36:32.567017: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8fc000 of size 1024 next 543 2025-03-17 22:36:32.567028: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8fc400 of size 1024 next 544 2025-03-17 22:36:32.567039: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8fc800 of size 1024 next 545 2025-03-17 22:36:32.567051: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8fcc00 of size 1024 next 546 2025-03-17 22:36:32.567062: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8fd000 of size 1024 next 547 2025-03-17 22:36:32.567074: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8fd400 of size 1024 next 550 2025-03-17 22:36:32.567085: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8fd800 of size 1024 next 551 2025-03-17 22:36:32.567096: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8fdc00 of size 1024 next 552 2025-03-17 22:36:32.567108: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8fe000 of size 1024 next 553 2025-03-17 22:36:32.567119: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8fe400 of size 1024 next 554 2025-03-17 22:36:32.567131: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8fe800 of size 4096 next 556 2025-03-17 22:36:32.567142: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c8ff800 of size 4096 next 557 2025-03-17 22:36:32.567153: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c900800 of size 4096 next 558 2025-03-17 22:36:32.567165: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c901800 of size 4096 next 559 2025-03-17 22:36:32.567176: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c902800 of size 4096 next 560 2025-03-17 22:36:32.567187: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c903800 of size 1024 next 561 2025-03-17 22:36:32.567199: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c903c00 of size 1024 next 562 2025-03-17 22:36:32.567210: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c904000 of size 1024 next 563 2025-03-17 22:36:32.567222: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c904400 of size 1024 next 564 2025-03-17 22:36:32.567233: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c904800 of size 1024 next 565 2025-03-17 22:36:32.567244: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c904c00 of size 1024 next 568 2025-03-17 22:36:32.567264: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c905000 of size 1024 next 569 2025-03-17 22:36:32.567275: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c905400 of size 1024 next 570 2025-03-17 22:36:32.567287: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c905800 of size 1024 next 571 2025-03-17 22:36:32.567298: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c905c00 of size 1024 next 572 2025-03-17 22:36:32.567310: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c906000 of size 4096 next 574 2025-03-17 22:36:32.567321: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c907000 of size 4096 next 575 2025-03-17 22:36:32.567333: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c908000 of size 4096 next 576 2025-03-17 22:36:32.567344: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c909000 of size 4096 next 577 2025-03-17 22:36:32.567355: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c90a000 of size 4096 next 578 2025-03-17 22:36:32.567367: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c90b000 of size 1024 next 579 2025-03-17 22:36:32.567378: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c90b400 of size 1024 next 580 2025-03-17 22:36:32.567390: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c90b800 of size 1024 next 581 2025-03-17 22:36:32.567401: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c90bc00 of size 1024 next 582 2025-03-17 22:36:32.567412: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c90c000 of size 1024 next 583 2025-03-17 22:36:32.567424: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c90c400 of size 1024 next 586 2025-03-17 22:36:32.567435: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c90c800 of size 1024 next 587 2025-03-17 22:36:32.567446: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c90cc00 of size 1024 next 588 2025-03-17 22:36:32.567458: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c90d000 of size 1024 next 589 2025-03-17 22:36:32.567469: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c90d400 of size 1024 next 590 2025-03-17 22:36:32.567480: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c90d800 of size 4096 next 592 2025-03-17 22:36:32.567492: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c90e800 of size 6144 next 112 2025-03-17 22:36:32.567504: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c910000 of size 524288 next 111 2025-03-17 22:36:32.567516: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c990000 of size 262144 next 135 2025-03-17 22:36:32.567527: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46c9d0000 of size 327680 next 124 2025-03-17 22:36:32.567539: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46ca20000 of size 589824 next 123 2025-03-17 22:36:32.567550: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cab0000 of size 262144 next 152 2025-03-17 22:36:32.567562: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46caf0000 of size 327680 next 142 2025-03-17 22:36:32.567574: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cb40000 of size 786432 next 18446744073709551615 2025-03-17 22:36:32.567585: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 8388608 2025-03-17 22:36:32.567597: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc00000 of size 4096 next 593 2025-03-17 22:36:32.567608: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc01000 of size 4096 next 594 2025-03-17 22:36:32.567619: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc02000 of size 4096 next 595 2025-03-17 22:36:32.567631: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc03000 of size 2048 next 598 2025-03-17 22:36:32.567650: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc03800 of size 2048 next 599 2025-03-17 22:36:32.567662: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc04000 of size 2048 next 600 2025-03-17 22:36:32.567673: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc04800 of size 2048 next 601 2025-03-17 22:36:32.567685: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc05000 of size 2048 next 602 2025-03-17 22:36:32.567696: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc05800 of size 256 next 605 2025-03-17 22:36:32.567708: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc05900 of size 256 next 606 2025-03-17 22:36:32.567719: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc05a00 of size 2048 next 604 2025-03-17 22:36:32.567730: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc06200 of size 2048 next 609 2025-03-17 22:36:32.567742: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc06a00 of size 2048 next 610 2025-03-17 22:36:32.567753: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc07200 of size 2048 next 611 2025-03-17 22:36:32.567765: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc07a00 of size 2048 next 612 2025-03-17 22:36:32.567776: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc08200 of size 256 next 614 2025-03-17 22:36:32.567788: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc08300 of size 256 next 615 2025-03-17 22:36:32.567799: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc08400 of size 8192 next 613 2025-03-17 22:36:32.567811: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc0a400 of size 8192 next 618 2025-03-17 22:36:32.567822: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc0c400 of size 8192 next 619 2025-03-17 22:36:32.567834: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc0e400 of size 8192 next 620 2025-03-17 22:36:32.567845: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc10400 of size 8192 next 621 2025-03-17 22:36:32.567856: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc12400 of size 256 next 622 2025-03-17 22:36:32.567868: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc12500 of size 256 next 623 2025-03-17 22:36:32.567879: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc12600 of size 8192 next 626 2025-03-17 22:36:32.567891: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc14600 of size 8192 next 627 2025-03-17 22:36:32.567902: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc16600 of size 8192 next 628 2025-03-17 22:36:32.567913: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc18600 of size 8192 next 629 2025-03-17 22:36:32.567925: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc1a600 of size 8192 next 630 2025-03-17 22:36:32.567936: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc1c600 of size 2048 next 631 2025-03-17 22:36:32.567948: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc1ce00 of size 2048 next 633 2025-03-17 22:36:32.567959: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc1d600 of size 2048 next 634 2025-03-17 22:36:32.567970: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc1de00 of size 2048 next 635 2025-03-17 22:36:32.567982: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc1e600 of size 2048 next 636 2025-03-17 22:36:32.567993: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fc46cc1ee00 of size 463360 next 160 2025-03-17 22:36:32.568005: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cc90000 of size 589824 next 158 2025-03-17 22:36:32.568029: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cd20000 of size 524288 next 171 2025-03-17 22:36:32.568040: I tensorflow/core/common_runtime/bfc_allocator.cc:990] Free at 7fc46cda0000 of size 1048576 next 222 2025-03-17 22:36:32.568052: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cea0000 of size 1048576 next 190 2025-03-17 22:36:32.568063: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46cfa0000 of size 1048576 next 189 2025-03-17 22:36:32.568075: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46d0a0000 of size 1048576 next 203 2025-03-17 22:36:32.568086: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46d1a0000 of size 1048576 next 251 2025-03-17 22:36:32.568098: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc46d2a0000 of size 1441792 next 18446744073709551615 2025-03-17 22:36:32.568109: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Next region of size 1048576 2025-03-17 22:36:32.568121: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e00000 of size 1280 next 1 2025-03-17 22:36:32.568133: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e00500 of size 256 next 5 2025-03-17 22:36:32.568144: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e00600 of size 256 next 8 2025-03-17 22:36:32.568156: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e00700 of size 256 next 9 2025-03-17 22:36:32.568167: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e00800 of size 256 next 10 2025-03-17 22:36:32.568179: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e00900 of size 256 next 11 2025-03-17 22:36:32.568190: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e00a00 of size 256 next 12 2025-03-17 22:36:32.568201: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e00b00 of size 256 next 13 2025-03-17 22:36:32.568213: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e00c00 of size 256 next 17 2025-03-17 22:36:32.568224: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e00d00 of size 256 next 19 2025-03-17 22:36:32.568235: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e00e00 of size 256 next 20 2025-03-17 22:36:32.568247: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e00f00 of size 256 next 21 2025-03-17 22:36:32.568258: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e01000 of size 256 next 22 2025-03-17 22:36:32.568269: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e01100 of size 256 next 24 2025-03-17 22:36:32.568281: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e01200 of size 256 next 25 2025-03-17 22:36:32.568292: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e01300 of size 256 next 23 2025-03-17 22:36:32.568303: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e01400 of size 256 next 28 2025-03-17 22:36:32.568315: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e01500 of size 256 next 29 2025-03-17 22:36:32.568326: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e01600 of size 256 next 30 2025-03-17 22:36:32.568337: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e01700 of size 256 next 31 2025-03-17 22:36:32.568349: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e01800 of size 256 next 33 2025-03-17 22:36:32.568360: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e01900 of size 256 next 34 2025-03-17 22:36:32.568372: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e01a00 of size 1024 next 32 2025-03-17 22:36:32.568383: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e01e00 of size 1024 next 37 2025-03-17 22:36:32.568403: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e02200 of size 1024 next 38 2025-03-17 22:36:32.568414: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e02600 of size 1024 next 39 2025-03-17 22:36:32.568426: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e02a00 of size 1024 next 40 2025-03-17 22:36:32.568437: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e02e00 of size 1024 next 41 2025-03-17 22:36:32.568449: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e03200 of size 1024 next 43 2025-03-17 22:36:32.568460: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e03600 of size 1024 next 44 2025-03-17 22:36:32.568471: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e03a00 of size 1024 next 45 2025-03-17 22:36:32.568483: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e03e00 of size 1024 next 46 2025-03-17 22:36:32.568494: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e04200 of size 256 next 48 2025-03-17 22:36:32.568506: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e04300 of size 256 next 49 2025-03-17 22:36:32.568517: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e04400 of size 256 next 50 2025-03-17 22:36:32.568528: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e04500 of size 256 next 51 2025-03-17 22:36:32.568540: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e04600 of size 256 next 52 2025-03-17 22:36:32.568551: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e04700 of size 256 next 53 2025-03-17 22:36:32.568562: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e04800 of size 256 next 56 2025-03-17 22:36:32.568574: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e04900 of size 256 next 57 2025-03-17 22:36:32.568585: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e04a00 of size 256 next 58 2025-03-17 22:36:32.568597: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e04b00 of size 256 next 14 2025-03-17 22:36:32.568608: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e04c00 of size 256 next 15 2025-03-17 22:36:32.568619: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e04d00 of size 256 next 16 2025-03-17 22:36:32.568631: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e04e00 of size 1024 next 60 2025-03-17 22:36:32.568642: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e05200 of size 1024 next 61 2025-03-17 22:36:32.568654: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e05600 of size 1024 next 62 2025-03-17 22:36:32.568665: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e05a00 of size 1024 next 63 2025-03-17 22:36:32.568676: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e05e00 of size 1024 next 64 2025-03-17 22:36:32.568688: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e06200 of size 256 next 66 2025-03-17 22:36:32.568699: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e06300 of size 256 next 67 2025-03-17 22:36:32.568711: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e06400 of size 256 next 68 2025-03-17 22:36:32.568722: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e06500 of size 256 next 69 2025-03-17 22:36:32.568733: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e06600 of size 256 next 70 2025-03-17 22:36:32.568745: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e06700 of size 256 next 73 2025-03-17 22:36:32.568756: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e06800 of size 256 next 74 2025-03-17 22:36:32.568767: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e06900 of size 256 next 75 2025-03-17 22:36:32.568788: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e06a00 of size 256 next 76 2025-03-17 22:36:32.568799: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e06b00 of size 256 next 77 2025-03-17 22:36:32.568811: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e06c00 of size 1024 next 79 2025-03-17 22:36:32.568822: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e07000 of size 1024 next 80 2025-03-17 22:36:32.568834: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e07400 of size 1024 next 81 2025-03-17 22:36:32.568845: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e07800 of size 1024 next 82 2025-03-17 22:36:32.568856: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e07c00 of size 1024 next 83 2025-03-17 22:36:32.568868: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e08000 of size 256 next 85 2025-03-17 22:36:32.568879: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e08100 of size 256 next 86 2025-03-17 22:36:32.568891: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e08200 of size 512 next 84 2025-03-17 22:36:32.568903: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e08400 of size 512 next 88 2025-03-17 22:36:32.568914: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e08600 of size 512 next 89 2025-03-17 22:36:32.568925: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e08800 of size 512 next 90 2025-03-17 22:36:32.568937: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e08a00 of size 512 next 91 2025-03-17 22:36:32.568948: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e08c00 of size 256 next 93 2025-03-17 22:36:32.568959: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e08d00 of size 256 next 94 2025-03-17 22:36:32.568971: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e08e00 of size 512 next 92 2025-03-17 22:36:32.568982: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e09000 of size 512 next 97 2025-03-17 22:36:32.568994: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e09200 of size 512 next 98 2025-03-17 22:36:32.569005: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e09400 of size 512 next 99 2025-03-17 22:36:32.569016: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e09600 of size 512 next 2 2025-03-17 22:36:32.569028: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e09800 of size 256 next 3 2025-03-17 22:36:32.569039: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e09900 of size 256 next 4 2025-03-17 22:36:32.569051: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e09a00 of size 16384 next 18 2025-03-17 22:36:32.569063: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e0da00 of size 256 next 100 2025-03-17 22:36:32.569074: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e0db00 of size 256 next 101 2025-03-17 22:36:32.569085: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e0dc00 of size 2048 next 104 2025-03-17 22:36:32.569097: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e0e400 of size 2048 next 105 2025-03-17 22:36:32.569108: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e0ec00 of size 2048 next 106 2025-03-17 22:36:32.569120: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e0f400 of size 2048 next 107 2025-03-17 22:36:32.569131: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e0fc00 of size 2048 next 108 2025-03-17 22:36:32.569142: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e10400 of size 256 next 109 2025-03-17 22:36:32.569154: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e10500 of size 256 next 110 2025-03-17 22:36:32.569173: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e10600 of size 2048 next 113 2025-03-17 22:36:32.569185: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e10e00 of size 2048 next 114 2025-03-17 22:36:32.569197: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e11600 of size 2048 next 115 2025-03-17 22:36:32.569208: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e11e00 of size 2048 next 116 2025-03-17 22:36:32.569219: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e12600 of size 2048 next 6 2025-03-17 22:36:32.569231: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e12e00 of size 37632 next 7 2025-03-17 22:36:32.569243: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e1c100 of size 512 next 118 2025-03-17 22:36:32.569254: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e1c300 of size 512 next 119 2025-03-17 22:36:32.569265: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e1c500 of size 512 next 120 2025-03-17 22:36:32.569277: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e1c700 of size 512 next 121 2025-03-17 22:36:32.569288: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e1c900 of size 512 next 122 2025-03-17 22:36:32.569300: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e1cb00 of size 512 next 125 2025-03-17 22:36:32.569311: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e1cd00 of size 512 next 126 2025-03-17 22:36:32.569322: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e1cf00 of size 512 next 127 2025-03-17 22:36:32.569334: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e1d100 of size 512 next 128 2025-03-17 22:36:32.569345: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e1d300 of size 512 next 129 2025-03-17 22:36:32.569356: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e1d500 of size 2048 next 130 2025-03-17 22:36:32.569368: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e1dd00 of size 2048 next 131 2025-03-17 22:36:32.569379: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e1e500 of size 2048 next 132 2025-03-17 22:36:32.569391: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e1ed00 of size 2048 next 133 2025-03-17 22:36:32.569402: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e1f500 of size 2048 next 134 2025-03-17 22:36:32.569413: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e1fd00 of size 512 next 136 2025-03-17 22:36:32.569425: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e1ff00 of size 512 next 137 2025-03-17 22:36:32.569436: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e20100 of size 512 next 138 2025-03-17 22:36:32.569447: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e20300 of size 512 next 139 2025-03-17 22:36:32.569459: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e20500 of size 512 next 140 2025-03-17 22:36:32.569470: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e20700 of size 512 next 141 2025-03-17 22:36:32.569482: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e20900 of size 512 next 143 2025-03-17 22:36:32.569493: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e20b00 of size 512 next 144 2025-03-17 22:36:32.569504: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e20d00 of size 512 next 145 2025-03-17 22:36:32.569516: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e20f00 of size 512 next 146 2025-03-17 22:36:32.569527: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e21100 of size 2048 next 147 2025-03-17 22:36:32.569547: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e21900 of size 2048 next 148 2025-03-17 22:36:32.569559: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e22100 of size 2048 next 149 2025-03-17 22:36:32.569570: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e22900 of size 2048 next 150 2025-03-17 22:36:32.569582: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e23100 of size 2048 next 151 2025-03-17 22:36:32.569593: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e23900 of size 512 next 153 2025-03-17 22:36:32.569605: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e23b00 of size 512 next 154 2025-03-17 22:36:32.569616: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e23d00 of size 512 next 155 2025-03-17 22:36:32.569627: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e23f00 of size 512 next 156 2025-03-17 22:36:32.569639: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e24100 of size 512 next 157 2025-03-17 22:36:32.569650: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e24300 of size 512 next 161 2025-03-17 22:36:32.569661: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e24500 of size 512 next 162 2025-03-17 22:36:32.569673: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e24700 of size 512 next 163 2025-03-17 22:36:32.569684: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e24900 of size 512 next 164 2025-03-17 22:36:32.569695: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e24b00 of size 512 next 165 2025-03-17 22:36:32.569707: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e24d00 of size 2048 next 166 2025-03-17 22:36:32.569718: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e25500 of size 2048 next 167 2025-03-17 22:36:32.569730: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e25d00 of size 2048 next 168 2025-03-17 22:36:32.569741: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e26500 of size 2048 next 169 2025-03-17 22:36:32.569752: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e26d00 of size 2048 next 170 2025-03-17 22:36:32.569764: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e27500 of size 1024 next 172 2025-03-17 22:36:32.569775: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e27900 of size 1024 next 173 2025-03-17 22:36:32.569787: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e27d00 of size 1024 next 174 2025-03-17 22:36:32.569798: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e28100 of size 1024 next 175 2025-03-17 22:36:32.569809: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e28500 of size 1024 next 176 2025-03-17 22:36:32.569821: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e28900 of size 256 next 178 2025-03-17 22:36:32.569832: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e28a00 of size 256 next 179 2025-03-17 22:36:32.569843: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e28b00 of size 1024 next 177 2025-03-17 22:36:32.569855: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e28f00 of size 1024 next 182 2025-03-17 22:36:32.569866: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e29300 of size 1024 next 183 2025-03-17 22:36:32.569878: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e29700 of size 1024 next 184 2025-03-17 22:36:32.569889: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e29b00 of size 1024 next 185 2025-03-17 22:36:32.569900: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e29f00 of size 256 next 187 2025-03-17 22:36:32.569919: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e2a000 of size 256 next 188 2025-03-17 22:36:32.569931: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e2a100 of size 4096 next 186 2025-03-17 22:36:32.569943: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e2b100 of size 4096 next 42 2025-03-17 22:36:32.569954: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e2c100 of size 65536 next 36 2025-03-17 22:36:32.569966: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e3c100 of size 65536 next 35 2025-03-17 22:36:32.569977: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e4c100 of size 4096 next 235 2025-03-17 22:36:32.569988: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e4d100 of size 4096 next 236 2025-03-17 22:36:32.570000: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e4e100 of size 4096 next 237 2025-03-17 22:36:32.570011: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e4f100 of size 4096 next 238 2025-03-17 22:36:32.570023: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e50100 of size 1024 next 239 2025-03-17 22:36:32.570034: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e50500 of size 1024 next 240 2025-03-17 22:36:32.570045: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e50900 of size 1024 next 241 2025-03-17 22:36:32.570057: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e50d00 of size 1024 next 242 2025-03-17 22:36:32.570068: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e51100 of size 1024 next 243 2025-03-17 22:36:32.570079: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e51500 of size 1024 next 246 2025-03-17 22:36:32.570091: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e51900 of size 1024 next 247 2025-03-17 22:36:32.570102: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e51d00 of size 1024 next 248 2025-03-17 22:36:32.570114: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e52100 of size 1024 next 249 2025-03-17 22:36:32.570125: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e52500 of size 1024 next 250 2025-03-17 22:36:32.570136: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e52900 of size 4096 next 252 2025-03-17 22:36:32.570148: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e53900 of size 4096 next 253 2025-03-17 22:36:32.570159: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e54900 of size 4096 next 254 2025-03-17 22:36:32.570170: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e55900 of size 4096 next 255 2025-03-17 22:36:32.570182: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e56900 of size 4096 next 256 2025-03-17 22:36:32.570193: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e57900 of size 1024 next 257 2025-03-17 22:36:32.570205: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e57d00 of size 1024 next 258 2025-03-17 22:36:32.570216: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e58100 of size 1024 next 259 2025-03-17 22:36:32.570227: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e58500 of size 1024 next 260 2025-03-17 22:36:32.570239: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e58900 of size 1024 next 261 2025-03-17 22:36:32.570250: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e58d00 of size 1024 next 264 2025-03-17 22:36:32.570262: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e59100 of size 1024 next 265 2025-03-17 22:36:32.570273: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e59500 of size 1024 next 266 2025-03-17 22:36:32.570284: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e59900 of size 1024 next 267 2025-03-17 22:36:32.570304: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e59d00 of size 1024 next 268 2025-03-17 22:36:32.570316: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e5a100 of size 4096 next 270 2025-03-17 22:36:32.570327: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e5b100 of size 4096 next 271 2025-03-17 22:36:32.570338: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e5c100 of size 4096 next 272 2025-03-17 22:36:32.570350: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e5d100 of size 4096 next 273 2025-03-17 22:36:32.570361: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e5e100 of size 4096 next 274 2025-03-17 22:36:32.570373: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e5f100 of size 1024 next 275 2025-03-17 22:36:32.570384: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e5f500 of size 1024 next 276 2025-03-17 22:36:32.570395: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e5f900 of size 1024 next 277 2025-03-17 22:36:32.570407: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e5fd00 of size 1024 next 278 2025-03-17 22:36:32.570418: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e60100 of size 1024 next 279 2025-03-17 22:36:32.570429: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e60500 of size 1024 next 282 2025-03-17 22:36:32.570441: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e60900 of size 1024 next 283 2025-03-17 22:36:32.570452: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e60d00 of size 1024 next 284 2025-03-17 22:36:32.570464: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e61100 of size 1024 next 285 2025-03-17 22:36:32.570475: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e61500 of size 1024 next 286 2025-03-17 22:36:32.570486: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e61900 of size 4096 next 288 2025-03-17 22:36:32.570498: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e62900 of size 6144 next 27 2025-03-17 22:36:32.570509: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e64100 of size 147456 next 26 2025-03-17 22:36:32.570521: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e88100 of size 65536 next 47 2025-03-17 22:36:32.570532: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2e98100 of size 65536 next 59 2025-03-17 22:36:32.570544: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2ea8100 of size 65536 next 65 2025-03-17 22:36:32.570556: I tensorflow/core/common_runtime/bfc_allocator.cc:990] InUse at 7fc5a2eb8100 of size 294656 next 18446744073709551615 2025-03-17 22:36:32.570567: I tensorflow/core/common_runtime/bfc_allocator.cc:995] Summary of in-use Chunks by size: 2025-03-17 22:36:32.570581: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 65 Chunks of size 256 totalling 16.2KiB 2025-03-17 22:36:32.570595: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 40 Chunks of size 512 totalling 20.0KiB 2025-03-17 22:36:32.570608: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 250 Chunks of size 1024 totalling 250.0KiB 2025-03-17 22:36:32.570621: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1280 totalling 1.2KiB 2025-03-17 22:36:32.570634: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 40 Chunks of size 2048 totalling 80.0KiB 2025-03-17 22:36:32.570647: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 117 Chunks of size 4096 totalling 468.0KiB 2025-03-17 22:36:32.570659: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 6144 totalling 12.0KiB 2025-03-17 22:36:32.570672: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 7680 totalling 7.5KiB 2025-03-17 22:36:32.570693: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 10 Chunks of size 8192 totalling 80.0KiB 2025-03-17 22:36:32.570706: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 16384 totalling 16.0KiB 2025-03-17 22:36:32.570719: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 37632 totalling 36.8KiB 2025-03-17 22:36:32.570732: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 6 Chunks of size 65536 totalling 384.0KiB 2025-03-17 22:36:32.570744: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 131072 totalling 128.0KiB 2025-03-17 22:36:32.570757: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 147456 totalling 432.0KiB 2025-03-17 22:36:32.570770: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 4 Chunks of size 262144 totalling 1.00MiB 2025-03-17 22:36:32.570782: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 294656 totalling 287.8KiB 2025-03-17 22:36:32.570795: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 327680 totalling 640.0KiB 2025-03-17 22:36:32.570807: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 524288 totalling 1.00MiB 2025-03-17 22:36:32.570820: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 3 Chunks of size 589824 totalling 1.69MiB 2025-03-17 22:36:32.570833: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 786432 totalling 768.0KiB 2025-03-17 22:36:32.570845: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 25 Chunks of size 1048576 totalling 25.00MiB 2025-03-17 22:36:32.570858: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 18 Chunks of size 1310720 totalling 22.50MiB 2025-03-17 22:36:32.570871: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1441792 totalling 1.38MiB 2025-03-17 22:36:32.570883: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 1835008 totalling 1.75MiB 2025-03-17 22:36:32.570931: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 2097152 totalling 4.00MiB 2025-03-17 22:36:32.570954: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 23 Chunks of size 2359296 totalling 51.75MiB 2025-03-17 22:36:32.570967: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 4194304 totalling 8.00MiB 2025-03-17 22:36:32.570980: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 8388608 totalling 8.00MiB 2025-03-17 22:36:32.570993: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 2 Chunks of size 9437184 totalling 18.00MiB 2025-03-17 22:36:32.571006: I tensorflow/core/common_runtime/bfc_allocator.cc:998] 1 Chunks of size 13500416 totalling 12.88MiB 2025-03-17 22:36:32.571018: I tensorflow/core/common_runtime/bfc_allocator.cc:1002] Sum Total of in-use chunks: 160.48MiB 2025-03-17 22:36:32.571030: I tensorflow/core/common_runtime/bfc_allocator.cc:1004] total_region_allocated_bytes_: 201195520 memory_limit_: 201195520 available bytes: 0 curr_region_allocation_bytes_: 268435456 2025-03-17 22:36:32.571048: I tensorflow/core/common_runtime/bfc_allocator.cc:1010] Stats: Limit: 201195520 InUse: 168275456 MaxInUse: 168275712 NumAllocs: 2113 MaxAllocSize: 13500416 2025-03-17 22:36:32.571076: W tensorflow/core/common_runtime/bfc_allocator.cc:439] __*****__******___**************x*********************************_************************_******** 2025-03-17 22:36:32.571134: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at cwise_ops_common.h:134 : Resource exhausted: OOM when allocating tensor with shape[3,3,512,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3473, 'mask_coco_origin', 16384, 25088, 'mask_coco_origin', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 3, 19, 10, 42, 21), datetime.datetime(2018, 3, 19, 10, 42, 21)) {'thcl': {'id': 454, 'mtr_user_id': 31, 'name': 'mask_coco_origin', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'photo_desc_type': 3473, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'], 'list_hashtags_csv': 'backgroud,person,bicycle,car,motorcycle,airplane,bus,train,truck,boat,trafficlight,firehydrant,stopsign,parkingmeter,bench,bird,cat,dog,horse,sheep,cow,elephant,bear,zebra,giraffe,backpack,umbrella,handbag,tie,suitcase,frisbee,skis,snowboard,sportsball,kite,baseballbat,baseballglove,skateboard,surfboard,tennisracket,bottle,wineglass,cup,fork,knife,spoon,bowl,banana,apple,sandwich,orange,broccoli,carrot,hotdog,pizza,donut,cake,chair,couch,pottedplant,bed,diningtable,toilet,tv,laptop,mouse,remote,keyboard,cellphone,microwave,oven,toaster,sink,refrigerator,book,clock,vase,scissors,teddybear,hairdrier,toothbrush', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 445, 'svm_hashtag_type_desc': 3473, 'photo_desc_type': 3473, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['backgroud', 'person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'trafficlight', 'firehydrant', 'stopsign', 'parkingmeter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sportsball', 'kite', 'baseballbat', 'baseballglove', 'skateboard', 'surfboard', 'tennisracket', 'bottle', 'wineglass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hotdog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'pottedplant', 'bed', 'diningtable', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cellphone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddybear', 'hairdrier', 'toothbrush'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_coco_origin NUM_CLASSES 81 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 Exception in mask_detect : OOM when allocating tensor with shape[3,3,512,512] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Add] we want to redo the detection Using TensorFlow backend. max_time_sub_proc : 3600 erreur pendant la detection Useless call to update_current_state in case -12 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : False eke 12-6-18 : saveMask need to be cleaned for new output ! ERROR : mask output needs to be a dictionnary now ! No output to save, continue without doing anything ! save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : -12 ERROR : 'int' object is not subscriptable reconnect to base ! warning , we can't find thcl infos in json_data warning , we can't find pdt infos in json_data #&_# TEST FAILED #&_# : tests/mask_test #&_# Error : invalid literal for int() with base 10: "'int' object is not subscriptable" /home/admin/workarea/git/Velours/python/tests/python_tests.py refs/heads/master_6419193983607b8ec28e462733ffc29d3101ac80 SQL :INSERT INTO MTRAdmin.monitor_sys (name, type, server, version_code, result_str, result_bool, lien , test_group ,test_name) VALUES ('python_test3','1','marlene','refs/heads/master_6419193983607b8ec28e462733ffc29d3101ac80','{"mask_detection": "fail"}','0','http://marlene.fotonower-preprod.com/job/2025/March/17032025/python_test3//data_2/data_log/job/2025/March/17032025/python_test3/log-python3----short_python3--v--marlene-21:35:01.txt','mask_detection','unknown'); #&_# END OF TEST #&_# : tests/mask_test #&_# #&_# BEGIN OF TEST : tests/datou_test #&_# /home/admin/workarea/git/Velours/python/tests/datou_test.py Datou All Test python version used : 3 ############################### TEST sam ################################ TEST SAM Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4573 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4573 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4573 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4573 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! no param json to modify List Step Type Loaded in datou : sam list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1189321094) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 1189321094 download finish for photo 1189321094 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.19778037071228027 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! WARNING : we have an input that is not a photo, we should get rid of it Calling datou_exec Inside datou_exec : verbose : True number of steps : 1 step1:sam Mon Mar 17 22:36:33 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1742247392_763596_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png': 1189321094} map_photo_id_path_extension : {1189321094: {'path': 'temp/1742247392_763596_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png', 'extension': 'png'}} map_subphoto_mainphoto : {} Beginning of datou step sam ! pht : 4677 Inside sam : nb paths : 1 (640, 960, 3) time for calcul the mask position with numpy : 0.0022253990173339844 nb_pixel_total : 5629 time to create 1 rle with old method : 0.006918191909790039 time for calcul the mask position with numpy : 0.0013988018035888672 nb_pixel_total : 13956 time to create 1 rle with old method : 0.0159604549407959 time for calcul the mask position with numpy : 0.0014281272888183594 nb_pixel_total : 3753 time to create 1 rle with old method : 0.004464149475097656 time for calcul the mask position with numpy : 0.0014781951904296875 nb_pixel_total : 16440 time to create 1 rle with old method : 0.020022153854370117 time for calcul the mask position with numpy : 0.0014901161193847656 nb_pixel_total : 10778 time to create 1 rle with old method : 0.013264656066894531 time for calcul the mask position with numpy : 0.002336263656616211 nb_pixel_total : 84025 time to create 1 rle with old method : 0.0969996452331543 time for calcul the mask position with numpy : 0.0013592243194580078 nb_pixel_total : 7587 time to create 1 rle with old method : 0.00923919677734375 time for calcul the mask position with numpy : 0.0013191699981689453 nb_pixel_total : 2940 time to create 1 rle with old method : 0.0036487579345703125 time for calcul the mask position with numpy : 0.001508951187133789 nb_pixel_total : 29451 time to create 1 rle with old method : 0.034174203872680664 time for calcul the mask position with numpy : 0.001348733901977539 nb_pixel_total : 2452 time to create 1 rle with old method : 0.0031328201293945312 time for calcul the mask position with numpy : 0.0013234615325927734 nb_pixel_total : 4291 time to create 1 rle with old method : 0.005124330520629883 time for calcul the mask position with numpy : 0.0015413761138916016 nb_pixel_total : 27691 time to create 1 rle with old method : 0.03238701820373535 time for calcul the mask position with numpy : 0.0013709068298339844 nb_pixel_total : 6639 time to create 1 rle with old method : 0.008357763290405273 time for calcul the mask position with numpy : 0.001367807388305664 nb_pixel_total : 3088 time to create 1 rle with old method : 0.004053354263305664 time for calcul the mask position with numpy : 0.001333475112915039 nb_pixel_total : 1227 time to create 1 rle with old method : 0.0015654563903808594 time for calcul the mask position with numpy : 0.0014414787292480469 nb_pixel_total : 9505 time to create 1 rle with old method : 0.011765003204345703 time for calcul the mask position with numpy : 0.0014107227325439453 nb_pixel_total : 3952 time to create 1 rle with old method : 0.004939556121826172 time for calcul the mask position with numpy : 0.0014946460723876953 nb_pixel_total : 16427 time to create 1 rle with old method : 0.019123554229736328 time for calcul the mask position with numpy : 0.0014336109161376953 nb_pixel_total : 3166 time to create 1 rle with old method : 0.003991603851318359 time for calcul the mask position with numpy : 0.0013535022735595703 nb_pixel_total : 2078 time to create 1 rle with old method : 0.0026314258575439453 time for calcul the mask position with numpy : 0.0015211105346679688 nb_pixel_total : 13001 time to create 1 rle with old method : 0.015635013580322266 time for calcul the mask position with numpy : 0.0013949871063232422 nb_pixel_total : 4283 time to create 1 rle with old method : 0.005454063415527344 time for calcul the mask position with numpy : 0.0014424324035644531 nb_pixel_total : 1647 time to create 1 rle with old method : 0.0021698474884033203 time for calcul the mask position with numpy : 0.0013442039489746094 nb_pixel_total : 906 time to create 1 rle with old method : 0.0012614727020263672 time for calcul the mask position with numpy : 0.0014421939849853516 nb_pixel_total : 5481 time to create 1 rle with old method : 0.006393909454345703 time for calcul the mask position with numpy : 0.0013322830200195312 nb_pixel_total : 1323 time to create 1 rle with old method : 0.0017616748809814453 time for calcul the mask position with numpy : 0.0014002323150634766 nb_pixel_total : 9932 time to create 1 rle with old method : 0.01189112663269043 time for calcul the mask position with numpy : 0.0015408992767333984 nb_pixel_total : 39061 time to create 1 rle with old method : 0.04579782485961914 time for calcul the mask position with numpy : 0.0014529228210449219 nb_pixel_total : 7687 time to create 1 rle with old method : 0.008904457092285156 time for calcul the mask position with numpy : 0.0014758110046386719 nb_pixel_total : 8610 time to create 1 rle with old method : 0.010363578796386719 time for calcul the mask position with numpy : 0.0014672279357910156 nb_pixel_total : 3533 time to create 1 rle with old method : 0.0043833255767822266 time for calcul the mask position with numpy : 0.0014700889587402344 nb_pixel_total : 11917 time to create 1 rle with old method : 0.014162063598632812 time for calcul the mask position with numpy : 0.001695871353149414 nb_pixel_total : 2381 time to create 1 rle with old method : 0.0031158924102783203 time for calcul the mask position with numpy : 0.001348257064819336 nb_pixel_total : 972 time to create 1 rle with old method : 0.001216888427734375 time for calcul the mask position with numpy : 0.0014410018920898438 nb_pixel_total : 2445 time to create 1 rle with old method : 0.0029342174530029297 time for calcul the mask position with numpy : 0.0013432502746582031 nb_pixel_total : 859 time to create 1 rle with old method : 0.0011763572692871094 time for calcul the mask position with numpy : 0.001386880874633789 nb_pixel_total : 2747 time to create 1 rle with old method : 0.0034589767456054688 time for calcul the mask position with numpy : 0.0013310909271240234 nb_pixel_total : 2815 time to create 1 rle with old method : 0.0033550262451171875 time for calcul the mask position with numpy : 0.001417398452758789 nb_pixel_total : 5406 time to create 1 rle with old method : 0.006545543670654297 time for calcul the mask position with numpy : 0.0014259815216064453 nb_pixel_total : 12999 time to create 1 rle with old method : 0.01597738265991211 time for calcul the mask position with numpy : 0.0015802383422851562 nb_pixel_total : 14626 time to create 1 rle with old method : 0.02138066291809082 time for calcul the mask position with numpy : 0.0015442371368408203 nb_pixel_total : 8706 time to create 1 rle with old method : 0.01068258285522461 time for calcul the mask position with numpy : 0.0014791488647460938 nb_pixel_total : 10616 time to create 1 rle with old method : 0.013424873352050781 time for calcul the mask position with numpy : 0.001405477523803711 nb_pixel_total : 1244 time to create 1 rle with old method : 0.0015041828155517578 time for calcul the mask position with numpy : 0.0014145374298095703 nb_pixel_total : 1025 time to create 1 rle with old method : 0.0013041496276855469 time for calcul the mask position with numpy : 0.0013854503631591797 nb_pixel_total : 3326 time to create 1 rle with old method : 0.0042307376861572266 time for calcul the mask position with numpy : 0.0013718605041503906 nb_pixel_total : 888 time to create 1 rle with old method : 0.0011677742004394531 time for calcul the mask position with numpy : 0.0014429092407226562 nb_pixel_total : 348 time to create 1 rle with old method : 0.0004982948303222656 time for calcul the mask position with numpy : 0.0013554096221923828 nb_pixel_total : 4126 time to create 1 rle with old method : 0.005117654800415039 time for calcul the mask position with numpy : 0.0014438629150390625 nb_pixel_total : 4176 time to create 1 rle with old method : 0.005315065383911133 time for calcul the mask position with numpy : 0.0014183521270751953 nb_pixel_total : 595 time to create 1 rle with old method : 0.0008237361907958984 time for calcul the mask position with numpy : 0.001430511474609375 nb_pixel_total : 3768 time to create 1 rle with old method : 0.004626274108886719 time for calcul the mask position with numpy : 0.0016522407531738281 nb_pixel_total : 39352 time to create 1 rle with old method : 0.04732108116149902 time for calcul the mask position with numpy : 0.0014357566833496094 nb_pixel_total : 577 time to create 1 rle with old method : 0.0007884502410888672 time for calcul the mask position with numpy : 0.0014672279357910156 nb_pixel_total : 16684 time to create 1 rle with old method : 0.02032780647277832 time for calcul the mask position with numpy : 0.0014095306396484375 nb_pixel_total : 2026 time to create 1 rle with old method : 0.0025854110717773438 time for calcul the mask position with numpy : 0.001443624496459961 nb_pixel_total : 2505 time to create 1 rle with old method : 0.0031108856201171875 time for calcul the mask position with numpy : 0.0013697147369384766 nb_pixel_total : 1672 time to create 1 rle with old method : 0.002207517623901367 time for calcul the mask position with numpy : 0.0014140605926513672 nb_pixel_total : 338 time to create 1 rle with old method : 0.0004913806915283203 time for calcul the mask position with numpy : 0.0013511180877685547 nb_pixel_total : 2422 time to create 1 rle with old method : 0.0031981468200683594 time for calcul the mask position with numpy : 0.0014276504516601562 nb_pixel_total : 693 time to create 1 rle with old method : 0.0009844303131103516 time for calcul the mask position with numpy : 0.0014202594757080078 nb_pixel_total : 1717 time to create 1 rle with old method : 0.0021805763244628906 time for calcul the mask position with numpy : 0.0014224052429199219 nb_pixel_total : 867 time to create 1 rle with old method : 0.001201629638671875 time for calcul the mask position with numpy : 0.0014293193817138672 nb_pixel_total : 2769 time to create 1 rle with old method : 0.003651142120361328 time for calcul the mask position with numpy : 0.0014371871948242188 nb_pixel_total : 1070 time to create 1 rle with old method : 0.0014426708221435547 time for calcul the mask position with numpy : 0.0014328956604003906 nb_pixel_total : 1208 time to create 1 rle with old method : 0.0015878677368164062 time for calcul the mask position with numpy : 0.0015294551849365234 nb_pixel_total : 13211 time to create 1 rle with old method : 0.016762495040893555 time for calcul the mask position with numpy : 0.0051615238189697266 nb_pixel_total : 18542 time to create 1 rle with old method : 0.024927854537963867 time for calcul the mask position with numpy : 0.0016460418701171875 nb_pixel_total : 1062 time to create 1 rle with old method : 0.0014209747314453125 time for calcul the mask position with numpy : 0.0013763904571533203 nb_pixel_total : 4450 time to create 1 rle with old method : 0.005469322204589844 time for calcul the mask position with numpy : 0.0013761520385742188 nb_pixel_total : 1009 time to create 1 rle with old method : 0.0013277530670166016 time for calcul the mask position with numpy : 0.001390218734741211 nb_pixel_total : 1750 time to create 1 rle with old method : 0.0021996498107910156 time for calcul the mask position with numpy : 0.0014503002166748047 nb_pixel_total : 8500 time to create 1 rle with old method : 0.010020017623901367 time for calcul the mask position with numpy : 0.0014240741729736328 nb_pixel_total : 1336 time to create 1 rle with old method : 0.0017139911651611328 time for calcul the mask position with numpy : 0.0014464855194091797 nb_pixel_total : 1518 time to create 1 rle with old method : 0.0018053054809570312 time for calcul the mask position with numpy : 0.0014541149139404297 nb_pixel_total : 9086 time to create 1 rle with old method : 0.011184215545654297 time for calcul the mask position with numpy : 0.0014519691467285156 nb_pixel_total : 267 time to create 1 rle with old method : 0.0003848075866699219 time for calcul the mask position with numpy : 0.0014226436614990234 nb_pixel_total : 970 time to create 1 rle with old method : 0.0012586116790771484 time for calcul the mask position with numpy : 0.0014646053314208984 nb_pixel_total : 713 time to create 1 rle with old method : 0.0009987354278564453 time for calcul the mask position with numpy : 0.0014035701751708984 nb_pixel_total : 614 time to create 1 rle with old method : 0.0008344650268554688 time for calcul the mask position with numpy : 0.0013647079467773438 nb_pixel_total : 583 time to create 1 rle with old method : 0.0008041858673095703 time for calcul the mask position with numpy : 0.0014295578002929688 nb_pixel_total : 1125 time to create 1 rle with old method : 0.0015130043029785156 time for calcul the mask position with numpy : 0.0014162063598632812 nb_pixel_total : 221 time to create 1 rle with old method : 0.00034427642822265625 time for calcul the mask position with numpy : 0.0014467239379882812 nb_pixel_total : 1501 time to create 1 rle with old method : 0.0018429756164550781 time for calcul the mask position with numpy : 0.001399993896484375 nb_pixel_total : 735 time to create 1 rle with old method : 0.0010912418365478516 time for calcul the mask position with numpy : 0.0013298988342285156 nb_pixel_total : 1634 time to create 1 rle with old method : 0.002051115036010742 time for calcul the mask position with numpy : 0.0014240741729736328 nb_pixel_total : 1413 time to create 1 rle with old method : 0.0019381046295166016 time for calcul the mask position with numpy : 0.0014147758483886719 nb_pixel_total : 300 time to create 1 rle with old method : 0.0004761219024658203 time for calcul the mask position with numpy : 0.0014476776123046875 nb_pixel_total : 7497 time to create 1 rle with old method : 0.00907444953918457 time for calcul the mask position with numpy : 0.001447916030883789 nb_pixel_total : 1438 time to create 1 rle with old method : 0.0019168853759765625 time for calcul the mask position with numpy : 0.0014188289642333984 nb_pixel_total : 249 time to create 1 rle with old method : 0.00037407875061035156 time for calcul the mask position with numpy : 0.0014176368713378906 nb_pixel_total : 595 time to create 1 rle with old method : 0.0008647441864013672 time for calcul the mask position with numpy : 0.0014286041259765625 nb_pixel_total : 1547 time to create 1 rle with old method : 0.0020074844360351562 time for calcul the mask position with numpy : 0.001424551010131836 nb_pixel_total : 917 time to create 1 rle with old method : 0.0013501644134521484 time for calcul the mask position with numpy : 0.0014309883117675781 nb_pixel_total : 2199 time to create 1 rle with old method : 0.0030150413513183594 time for calcul the mask position with numpy : 0.001445770263671875 nb_pixel_total : 949 time to create 1 rle with old method : 0.001325845718383789 time for calcul the mask position with numpy : 0.0014178752899169922 nb_pixel_total : 956 time to create 1 rle with old method : 0.0013213157653808594 time for calcul the mask position with numpy : 0.001428365707397461 nb_pixel_total : 2657 time to create 1 rle with old method : 0.003580331802368164 time for calcul the mask position with numpy : 0.0014328956604003906 nb_pixel_total : 828 time to create 1 rle with old method : 0.0012030601501464844 insert ignore into MTRPhoto.crop_hashtag_ids (photo_id, hashtag_id, `type`,x0,x1,y0,y1,score) VALUES (%s,%s,%s,%s,%s,%s,%s,%s) batch 1 Loaded 99 chid ids of type : 4677 Number RLEs to save : 9418 INSERT IGNORE INTO MTRPhoto.crop_segments (`crop_hashtag_id`, `x0`, `y0`, `length`) VALUES (%s, %s, %s , %s) first line : ('3716815690', '464', '201', '4') ... last line : ('3716815788', '815', '44', '5') INSERT IGNORE INTO MTRPhoto.crop_sum_segments (`crop_hashtag_id`, `sum_segments`) VALUES (%s, %s) TO DO : save crop sub photo not yet done ! After datou_step_exec type output : map_portfolio_photo : len 0 keys : dict_keys([]) Inside saveOutput : final : True verbose : True saveOutput not yet implemented for datou_step.type : sam we use saveGeneral [1189321094] map_info['map_portfolio_photo'] : {} final : True mtd_id 4573 list_pids : [1189321094] Looping around the photos to save general results len do output : 1 /1189321094Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4573', None, None, None, None, None, None, None, None) ('4573', None, '1189321094', None, None, None, None, None, None) begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4573', None, '1189321094', 'None', None, None, None, None, None)] time used for this insertion : 0.012363433837890625 save_final save missing photos in datou_result : time spend for datou_step_exec : 9.952963829040527 time spend to save output : 0.01257634162902832 total time spend for step 1 : 9.965540170669556 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {'1189321094': [[, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ], 'temp/1742247392_763596_1189321094_9626af7f95d010f2a4fd524688d4ea22_76896585.png']} nb_objects detect : 99 ############################### TEST frcnn ################################ test frcnn Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4184 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4184 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4184 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4184 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! no param json to modify List Step Type Loaded in datou : frcnn list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (917754606) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 917754606 download finish for photo 917754606 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.13259005546569824 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 1 step1:frcnn Mon Mar 17 22:36:43 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1742247403_763596_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg': 917754606} map_photo_id_path_extension : {917754606: {'path': 'temp/1742247403_763596_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou step Faster rcnn ! classes : ['background', 'plaque'] pht : 4370 caffemodel_name (should be vgg16_immat_307 but not used because net loaded outside in the fonction) : {'id': 3375, 'mtr_user_id': 31, 'name': 'detection_plaque_valcor_010622', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,plaque', 'svm_portfolios_learning': '0,0', 'photo_hashtag_type': 4370, 'photo_desc_type': 5676, 'type_classification': 'caffe_faster_rcnn', 'hashtag_id_list': '0,0'} To loadFromThcl() model_param file didn't exist model_name : detection_plaque_valcor_010622 model_type : caffe_faster_rcnn list file need : ['caffemodel', 'test.prototxt'] file exist in s3 : ['caffemodel', 'test.prototxt'] file manque in s3 : [] local folder : /data/models_weight/detection_plaque_valcor_010622 /data/models_weight/detection_plaque_valcor_010622/caffemodel size_local : 349723073 size in s3 : 349723073 create time local : 2022-07-12 14:12:27 create time in s3 : 2022-06-01 15:05:56 caffemodel already exist and didn't need to update /data/models_weight/detection_plaque_valcor_010622/test.prototxt size_local : 7163 size in s3 : 7163 create time local : 2022-07-12 14:12:27 create time in s3 : 2022-06-01 15:05:55 test.prototxt already exist and didn't need to update prototxt : /data/models_weight/detection_plaque_valcor_010622/test.prototxt caffemodel : /data/models_weight/detection_plaque_valcor_010622/caffemodel Loaded network /data/models_weight/detection_plaque_valcor_010622/caffemodel About to compute detect_faster_rcnn : len(args) : 1 Inside frcnn step exec : nb paths : 1 image_path : temp/1742247403_763596_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg image_size (600, 800, 3) [[[ 4 6 6] [ 5 7 7] [ 6 8 8] ... [207 215 214] [206 214 213] [206 214 213]] [[ 4 6 6] [ 5 7 7] [ 6 8 8] ... [207 215 214] [206 214 213] [206 214 213]] [[ 4 6 6] [ 5 7 7] [ 6 8 8] ... [207 215 214] [206 214 213] [206 214 213]] ... [[ 14 16 16] [ 13 15 15] [ 11 13 13] ... [198 206 205] [198 206 205] [198 206 205]] [[ 16 18 18] [ 14 16 16] [ 11 13 13] ... [206 214 213] [206 214 213] [206 214 213]] [[ 13 15 15] [ 12 14 14] [ 9 11 11] ... [210 218 217] [210 218 217] [210 218 217]]] Detection took 0.077s for 300 object proposals c : plaque list_crops.shape (72, 5) proba : 0.063799664 (374.12717, 293.91925, 430.81247, 317.80878) proba : 0.052248564 (382.18066, 297.18484, 552.365, 344.6536) proba : 0.012269895 (345.35458, 272.42722, 468.85025, 320.72153) We are managing local photo_id len de result frcnn : 1 After datou_step_exec type output : time spend for datou_step_exec : 2.2439355850219727 time spend to save output : 8.273124694824219e-05 total time spend for step 1 : 2.244018316268921 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : True Inside saveFrcnn : final : True verbose : True threshold to save the result : 0.1 output flattener : [(0, 493029425, 4370, 374, 430, 293, 317, 0.063799664, None), (0, 493029425, 4370, 382, 552, 297, 344, 0.052248564, None), (0, 493029425, 4370, 345, 468, 272, 320, 0.012269895, None)] Warning : no hashtag_ids to insert in the database final : True begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4184', None, '917754606', '0', 0, '0', 493061979, '0', None)] time used for this insertion : 0.012545347213745117 [917754606] map_info['map_portfolio_photo'] : {} final : True mtd_id 4184 list_pids : [917754606] Looping around the photos to save general results len do output : 1 /0 before output type Managing all output in save final without adding information in the mtr_datou_result ('4184', None, None, None, None, None, None, None, None) ('4184', None, '917754606', None, None, None, None, None, None) begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4184', None, '917754606', None, None, None, None, None, None)] time used for this insertion : 0.020596981048583984 save_final save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {0: [[(0, 493029425, 4370, 374, 430, 293, 317, 0.063799664, None), (0, 493029425, 4370, 382, 552, 297, 344, 0.052248564, None), (0, 493029425, 4370, 345, 468, 272, 320, 0.012269895, None)], 'temp/1742247403_763596_917754606_35f3c9ae49686a6be16030c6ec25c9ee.jpg']} ############################### TEST thcl ################################ TEST THCL Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=2 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=2 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 2 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=2 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : step 1 thcl is not linked in the step_by_step architecture ! WARNING : step 2 argmax is not linked in the step_by_step architecture ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! no param json to modify List Step Type Loaded in datou : thcl, argmax list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (916235064) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 916235064 download finish for photo 916235064 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.12566709518432617 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 2 step1:thcl Mon Mar 17 22:36:45 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1742247405_763596_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg': 916235064} map_photo_id_path_extension : {916235064: {'path': 'temp/1742247405_763596_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou step Thcl ! multi_thcl or not :False multi_thcl_cond or not :False dic_thcl : {'355': 1} we are using the classfication for only one thcl 355 In convert_file_to_np l 337 : 1 l343 1 l357 after caffe.io.load_image dimension du image : (3, (66, 66, 3)) dimension displayed ! time to import caffe and check if the image exist : 0.005656242370605469 time to convert the images to numpy array : 0.0034570693969726562 total time to convert the images to numpy array : 0.009366512298583984 list photo_ids error: [] list photo_ids correct : [916235064] number of photos to traite : 1 try to delete the photos incorrect in DB tagging for thcl : 355 To do loadFromThcl(), then load ParamDescType : thcl355 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (355) thcls : [{'id': 355, 'mtr_user_id': 31, 'name': 'car_360_1027', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'c_elysee_1027_gao__port_506302,mokka_1027_gao__port_506374,captur_1027_gao__port_506399,sorento_1027_gao__port_506192,navara_1027_gao__port_506205,xc90_1027_gao__port_506350,saxo_1027_gao__port_506052,trafic_1027_gao__port_506295,punto_evo_1027_gao__port_506066,5_1027_gao__port_506117,250_1027_gao__port_506065,d_max_1027_gao__port_506125,panamera_1027_gao__port_506387,alhambra_1027_gao__port_506381,x6_1027_gao__port_506349,vitara_1027_gao__port_506328,fiesta_1027_gao__port_506377,qashqai_1027_gao__port_506286,147_1027_gao__port_506124,c5_1027_gao__port_506172,q5_1027_gao__port_506206,giulia_1027_gao__port_506178,karl_1027_gao__port_506371,mehari_1027_gao__port_506076,911_1027_gao__port_506114,508_1027_gao__port_506329,idea_1027_gao__port_506122,megane_1027_gao__port_506220,ghibli_1027_gao__port_506174,touareg_1027_gao__port_506224,i10_1027_gao__port_506232,jumper_1027_gao__port_506234,classe_clk_1027_gao__port_506173,kuga_1027_gao__port_506181,ct_1027_gao__port_506323,leon_1027_gao__port_506326,ds5_1027_gao__port_506376,cordoba_1027_gao__port_506048,classe_cla_1027_gao__port_506400,jumpy_1027_gao__port_506179,avensis_1027_gao__port_506311,juke_1027_gao__port_506325,4008_1027_gao__port_506402,190_series_1027_gao__port_506051,serie_3_1027_gao__port_506294,q7_1027_gao__port_506318,glc_1027_gao__port_506303,grand_vitara_1027_gao__port_506175,s40_1027_gao__port_506099,toledo_1027_gao__port_506061,5008_1027_gao__port_506337,continental_1027_gao__port_506250,coupe_1027_gao__port_506082,iq_1027_gao__port_506166,407_1027_gao__port_506133,touran_1027_gao__port_506308,300c_1027_gao__port_506078,classe_gl_1027_gao__port_506340,vivaro_1027_gao__port_506310,sl_1027_gao__port_506100,elise_1027_gao__port_506121,1007_1027_gao__port_506070,i40_1027_gao__port_506218,bipper_tepee_1027_gao__port_506227,focus_1027_gao__port_506272,primera_1027_gao__port_506147,r4_1027_gao__port_506160,a8_1027_gao__port_506265,boxer_1027_gao__port_506202,s5_1027_gao__port_506222,r21_1027_gao__port_506093,c3_1027_gao__port_506257,santa_fe_1027_gao__port_506208,m4_1027_gao__port_506344,safrane_1027_gao__port_506077,classe_gle_1027_gao__port_506395,0_1027_gao__port_506094,ix35_1027_gao__port_506219,carens_1027_gao__port_506298,classe_a_1027_gao__port_506339,ix20_1027_gao__port_506343,note_1027_gao__port_506365,a5_1027_gao__port_506200,sx4_1027_gao__port_506348,sandero_1027_gao__port_506198,3008_1027_gao__port_506385,q50_1027_gao__port_506239,latitude_1027_gao__port_506236,v40_1027_gao__port_506391,xsara_1027_gao__port_506087,grand_c_max_1027_gao__port_506342,swift_1027_gao__port_506149,serie_1_1027_gao__port_506184,xc70_1027_gao__port_506393,master_1027_gao__port_506203,clio_1027_gao__port_506280,duster_1027_gao__port_506216,traveller_1027_gao__port_506403,tipo_1027_gao__port_506355,rav_4_1027_gao__port_506332,coccinelle_1027_gao__port_506259,spacetourer_1027_gao__port_506401,xe_1027_gao__port_506357,ds3_1027_gao__port_506324,mx_5_1027_gao__port_506098,land_cruiser_1027_gao__port_506315,classe_b_1027_gao__port_506335,806_1027_gao__port_506088,rx_8_1027_gao__port_506046,spark_1027_gao__port_506185,6_1027_gao__port_506171,bravo_1027_gao__port_506080,nx_1027_gao__port_506345,sharan_1027_gao__port_506347,x_type_1027_gao__port_506067,jimny_1027_gao__port_506233,wrangler_1027_gao__port_506225,c_crosser_1027_gao__port_506312,v70_1027_gao__port_506278,classe_e_1027_gao__port_506300,classe_v_1027_gao__port_506258,m3_1027_gao__port_506182,abarth_500_1027_gao__port_506226,serie_6_1027_gao__port_506262,modus_1027_gao__port_506146,3_1027_gao__port_506113,405_1027_gao__port_506108,allroad_1027_gao__port_506297,auris_1027_gao__port_506322,galaxy_1027_gao__port_506143,giulietta_1027_gao__port_506363,106_1027_gao__port_506073,classe_m_1027_gao__port_506154,espace_1027_gao__port_506313,panda_1027_gao__port_506189,rcz_1027_gao__port_506197,4007_1027_gao__port_506162,classe_cl_1027_gao__port_506249,leaf_1027_gao__port_506139,octavia_1027_gao__port_506237,ds4_1027_gao__port_506336,freelander_1027_gao__port_506084,evasion_1027_gao__port_506109,punto_1027_gao__port_506106,2cv_1027_gao__port_506045,x4_1027_gao__port_506392,antara_1027_gao__port_506247,murano_1027_gao__port_506316,alto_1027_gao__port_506201,meriva_1027_gao__port_506353,orlando_1027_gao__port_506305,new_beetle_1027_gao__port_506050,306_1027_gao__port_506145,tiguan_1027_gao__port_506362,s_type_1027_gao__port_506101,c1_1027_gao__port_506128,vectra_1027_gao__port_506044,outlander_1027_gao__port_506317,307_1027_gao__port_506074,a6_s6_1027_gao__port_506134,nemo_combi_1027_gao__port_506196,berlingo_1027_gao__port_506194,partner_1027_gao__port_506285,cayenne_1027_gao__port_506177,quattroporte_1027_gao__port_506240,c_max_1027_gao__port_506282,fabia_1027_gao__port_506396,cx_3_1027_gao__port_506281,x_trail_1027_gao__port_506264,scirocco_1027_gao__port_506276,matiz_1027_gao__port_506144,tigra_1027_gao__port_506069,escort_1027_gao__port_506091,c2_1027_gao__port_506081,mini_1027_gao__port_506168,i30_1027_gao__port_506291,picanto_1027_gao__port_506238,mito_1027_gao__port_506072,impreza_1027_gao__port_506085,kangoo_1027_gao__port_506235,a4_1027_gao__port_506193,cayman_1027_gao__port_506268,sportage_1027_gao__port_506148,up_1027_gao__port_506356,optima_1027_gao__port_506386,defender_1027_gao__port_506229,serie_2_1027_gao__port_506256,edge_1027_gao__port_506187,r19_1027_gao__port_506110,jetta_1027_gao__port_506304,eos_1027_gao__port_506115,accord_1027_gao__port_506214,yaris_1027_gao__port_506334,classe_cls_1027_gao__port_506289,polo_1027_gao__port_506361,serie_4_1027_gao__port_506366,mini_cabriolet_1027_gao__port_506204,prius_1027_gao__port_506190,lodgy_1027_gao__port_506188,serie_7_1027_gao__port_506307,c15_1027_gao__port_506055,kadjar_1027_gao__port_506389,insignia_1027_gao__port_506364,308_1027_gao__port_506279,roomster_1027_gao__port_506241,80_1027_gao__port_506057,309_1027_gao__port_506063,tucson_1027_gao__port_506320,x3_1027_gao__port_506212,xf_1027_gao__port_506263,2008_1027_gao__port_506394,passat_1027_gao__port_506306,compass_1027_gao__port_506260,twingo_1027_gao__port_506309,micra_1027_gao__port_506221,golf_1027_gao__port_506155,soul_1027_gao__port_506176,rapid_1027_gao__port_506398,forester_1027_gao__port_506360,slk_1027_gao__port_506210,forfour_1027_gao__port_506341,serie_5_1027_gao__port_506209,xj_1027_gao__port_506170,pajero_1027_gao__port_506097,agila_1027_gao__port_506119,a6_1027_gao__port_506163,fox_1027_gao__port_506092,boxster_1027_gao__port_506267,altea_1027_gao__port_506246,samurai_1027_gao__port_506047,trax_1027_gao__port_506296,getz_1027_gao__port_506058,cherokee_1027_gao__port_506269,koleos_1027_gao__port_506378,z_series_1027_gao__port_506123,ecosport_1027_gao__port_506271,space_star_1027_gao__port_506277,rs3_sportback_1027_gao__port_506207,civic_1027_gao__port_506141,talisman_1027_gao__port_506390,f_pace_1027_gao__port_506314,classe_c_1027_gao__port_506299,tt_1027_gao__port_506075,pathfinder_1027_gao__port_506183,156_1027_gao__port_506157,cx_5_1027_gao__port_506228,scenic_1027_gao__port_506255,yeti_1027_gao__port_506358,mustang_1027_gao__port_506053,stilo_1027_gao__port_506060,ateca_1027_gao__port_506382,fiorino_1027_gao__port_506217,classe_glk_1027_gao__port_506290,fortwo_1027_gao__port_506230,cruze_1027_gao__port_506186,107_1027_gao__port_506213,aygo_1027_gao__port_506248,rx_1027_gao__port_506354,500_1027_gao__port_506245,bora_1027_gao__port_506104,transit_1027_gao__port_506111,pt_cruiser_1027_gao__port_506054,patrol_1027_gao__port_506068,r8_1027_gao__port_506156,xm_1027_gao__port_506102,s60_1027_gao__port_506191,aveo_1027_gao__port_506158,captiva_1027_gao__port_506159,ax_1027_gao__port_506153,rexton_1027_gao__port_506107,camaro_1027_gao__port_506056,ypsilon_1027_gao__port_506131,delta_1027_gao__port_506165,c4_1027_gao__port_506370,zx_1027_gao__port_506161,verso_1027_gao__port_506242,superb_1027_gao__port_506327,r5_1027_gao__port_506253,caddy_1027_gao__port_506330,x5_1027_gao__port_506243,f_type_1027_gao__port_506231,fusion_1027_gao__port_506096,dokker_1027_gao__port_506331,205_1027_gao__port_506062,macan_1027_gao__port_506195,tourneo_1027_gao__port_506369,108_1027_gao__port_506384,9_3_1027_gao__port_506071,mondeo_1027_gao__port_506116,cr_v_1027_gao__port_506164,c30_1027_gao__port_506090,pulsar_1027_gao__port_506397,ibiza_1027_gao__port_506273,a1_1027_gao__port_506338,matrix_1027_gao__port_506140,carnival_1027_gao__port_506136,xantia_1027_gao__port_506086,terrano_1027_gao__port_506083,q3_1027_gao__port_506275,hr_v_1027_gao__port_506283,expert_1027_gao__port_506142,multivan_1027_gao__port_506383,venga_1027_gao__port_506380,scudo_1027_gao__port_506129,laguna_1027_gao__port_506368,vel_satis_1027_gao__port_506130,b_max_1027_gao__port_506367,ignis_1027_gao__port_506292,159_1027_gao__port_506064,grande_punto_1027_gao__port_506138,logan_1027_gao__port_506167,s_max_1027_gao__port_506223,caravelle_1027_gao__port_506351,adam_1027_gao__port_506079,406_1027_gao__port_506132,q30_1027_gao__port_506293,almera_1027_gao__port_506089,corsa_1027_gao__port_506095,corolla_1027_gao__port_506120,xc60_1027_gao__port_506388,viano_1027_gao__port_506211,pro_cee_d_1027_gao__port_506274,a3_1027_gao__port_506321,v50_1027_gao__port_506150,voyager_1027_gao__port_506169,corvette_1027_gao__port_506049,rio_1027_gao__port_506379,jazz_1027_gao__port_506252,200_1027_gao__port_506112,tts_1027_gao__port_506199,zafira_1027_gao__port_506287,asx_1027_gao__port_506266,607_1027_gao__port_506118,207_1027_gao__port_506103,classe_s_1027_gao__port_506301,c6_1027_gao__port_506105,express_1027_gao__port_506137,classe_gla_1027_gao__port_506352,v60_1027_gao__port_506333,ka_1027_gao__port_506180,range_rover_1027_gao__port_506254,discovery_1027_gao__port_506375,classe_r_1027_gao__port_506270,transporter_1027_gao__port_506319,cee_d_1027_gao__port_506288,zoe_1027_gao__port_506244,i20_1027_gao__port_506284,gtv_1027_gao__port_506059,s4_avant_1027_gao__port_506261,x1_1027_gao__port_506372,autres_1027_gao__port_506127,208_1027_gao__port_506359,c8_1027_gao__port_506135,astra_1027_gao__port_506215,2_1027_gao__port_506151,doblo_1027_gao__port_506251,807_1027_gao__port_506152,206_1027_gao__port_506126,a7_1027_gao__port_506373,renegade_1027_gao__port_506346', 'svm_portfolios_learning': '506302,506374,506399,506192,506205,506350,506052,506295,506066,506117,506065,506125,506387,506381,506349,506328,506377,506286,506124,506172,506206,506178,506371,506076,506114,506329,506122,506220,506174,506224,506232,506234,506173,506181,506323,506326,506376,506048,506400,506179,506311,506325,506402,506051,506294,506318,506303,506175,506099,506061,506337,506250,506082,506166,506133,506308,506078,506340,506310,506100,506121,506070,506218,506227,506272,506147,506160,506265,506202,506222,506093,506257,506208,506344,506077,506395,506094,506219,506298,506339,506343,506365,506200,506348,506198,506385,506239,506236,506391,506087,506342,506149,506184,506393,506203,506280,506216,506403,506355,506332,506259,506401,506357,506324,506098,506315,506335,506088,506046,506185,506171,506080,506345,506347,506067,506233,506225,506312,506278,506300,506258,506182,506226,506262,506146,506113,506108,506297,506322,506143,506363,506073,506154,506313,506189,506197,506162,506249,506139,506237,506336,506084,506109,506106,506045,506392,506247,506316,506201,506353,506305,506050,506145,506362,506101,506128,506044,506317,506074,506134,506196,506194,506285,506177,506240,506282,506396,506281,506264,506276,506144,506069,506091,506081,506168,506291,506238,506072,506085,506235,506193,506268,506148,506356,506386,506229,506256,506187,506110,506304,506115,506214,506334,506289,506361,506366,506204,506190,506188,506307,506055,506389,506364,506279,506241,506057,506063,506320,506212,506263,506394,506306,506260,506309,506221,506155,506176,506398,506360,506210,506341,506209,506170,506097,506119,506163,506092,506267,506246,506047,506296,506058,506269,506378,506123,506271,506277,506207,506141,506390,506314,506299,506075,506183,506157,506228,506255,506358,506053,506060,506382,506217,506290,506230,506186,506213,506248,506354,506245,506104,506111,506054,506068,506156,506102,506191,506158,506159,506153,506107,506056,506131,506165,506370,506161,506242,506327,506253,506330,506243,506231,506096,506331,506062,506195,506369,506384,506071,506116,506164,506090,506397,506273,506338,506140,506136,506086,506083,506275,506283,506142,506383,506380,506129,506368,506130,506367,506292,506064,506138,506167,506223,506351,506079,506132,506293,506089,506095,506120,506388,506211,506274,506321,506150,506169,506049,506379,506252,506112,506199,506287,506266,506118,506103,506301,506105,506137,506352,506333,506180,506254,506375,506270,506319,506288,506244,506284,506059,506261,506372,506127,506359,506135,506215,506151,506251,506152,506126,506373,506346', 'photo_hashtag_type': 332, 'photo_desc_type': 3390, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 355, 'mtr_user_id': 31, 'name': 'car_360_1027', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'c_elysee_1027_gao__port_506302,mokka_1027_gao__port_506374,captur_1027_gao__port_506399,sorento_1027_gao__port_506192,navara_1027_gao__port_506205,xc90_1027_gao__port_506350,saxo_1027_gao__port_506052,trafic_1027_gao__port_506295,punto_evo_1027_gao__port_506066,5_1027_gao__port_506117,250_1027_gao__port_506065,d_max_1027_gao__port_506125,panamera_1027_gao__port_506387,alhambra_1027_gao__port_506381,x6_1027_gao__port_506349,vitara_1027_gao__port_506328,fiesta_1027_gao__port_506377,qashqai_1027_gao__port_506286,147_1027_gao__port_506124,c5_1027_gao__port_506172,q5_1027_gao__port_506206,giulia_1027_gao__port_506178,karl_1027_gao__port_506371,mehari_1027_gao__port_506076,911_1027_gao__port_506114,508_1027_gao__port_506329,idea_1027_gao__port_506122,megane_1027_gao__port_506220,ghibli_1027_gao__port_506174,touareg_1027_gao__port_506224,i10_1027_gao__port_506232,jumper_1027_gao__port_506234,classe_clk_1027_gao__port_506173,kuga_1027_gao__port_506181,ct_1027_gao__port_506323,leon_1027_gao__port_506326,ds5_1027_gao__port_506376,cordoba_1027_gao__port_506048,classe_cla_1027_gao__port_506400,jumpy_1027_gao__port_506179,avensis_1027_gao__port_506311,juke_1027_gao__port_506325,4008_1027_gao__port_506402,190_series_1027_gao__port_506051,serie_3_1027_gao__port_506294,q7_1027_gao__port_506318,glc_1027_gao__port_506303,grand_vitara_1027_gao__port_506175,s40_1027_gao__port_506099,toledo_1027_gao__port_506061,5008_1027_gao__port_506337,continental_1027_gao__port_506250,coupe_1027_gao__port_506082,iq_1027_gao__port_506166,407_1027_gao__port_506133,touran_1027_gao__port_506308,300c_1027_gao__port_506078,classe_gl_1027_gao__port_506340,vivaro_1027_gao__port_506310,sl_1027_gao__port_506100,elise_1027_gao__port_506121,1007_1027_gao__port_506070,i40_1027_gao__port_506218,bipper_tepee_1027_gao__port_506227,focus_1027_gao__port_506272,primera_1027_gao__port_506147,r4_1027_gao__port_506160,a8_1027_gao__port_506265,boxer_1027_gao__port_506202,s5_1027_gao__port_506222,r21_1027_gao__port_506093,c3_1027_gao__port_506257,santa_fe_1027_gao__port_506208,m4_1027_gao__port_506344,safrane_1027_gao__port_506077,classe_gle_1027_gao__port_506395,0_1027_gao__port_506094,ix35_1027_gao__port_506219,carens_1027_gao__port_506298,classe_a_1027_gao__port_506339,ix20_1027_gao__port_506343,note_1027_gao__port_506365,a5_1027_gao__port_506200,sx4_1027_gao__port_506348,sandero_1027_gao__port_506198,3008_1027_gao__port_506385,q50_1027_gao__port_506239,latitude_1027_gao__port_506236,v40_1027_gao__port_506391,xsara_1027_gao__port_506087,grand_c_max_1027_gao__port_506342,swift_1027_gao__port_506149,serie_1_1027_gao__port_506184,xc70_1027_gao__port_506393,master_1027_gao__port_506203,clio_1027_gao__port_506280,duster_1027_gao__port_506216,traveller_1027_gao__port_506403,tipo_1027_gao__port_506355,rav_4_1027_gao__port_506332,coccinelle_1027_gao__port_506259,spacetourer_1027_gao__port_506401,xe_1027_gao__port_506357,ds3_1027_gao__port_506324,mx_5_1027_gao__port_506098,land_cruiser_1027_gao__port_506315,classe_b_1027_gao__port_506335,806_1027_gao__port_506088,rx_8_1027_gao__port_506046,spark_1027_gao__port_506185,6_1027_gao__port_506171,bravo_1027_gao__port_506080,nx_1027_gao__port_506345,sharan_1027_gao__port_506347,x_type_1027_gao__port_506067,jimny_1027_gao__port_506233,wrangler_1027_gao__port_506225,c_crosser_1027_gao__port_506312,v70_1027_gao__port_506278,classe_e_1027_gao__port_506300,classe_v_1027_gao__port_506258,m3_1027_gao__port_506182,abarth_500_1027_gao__port_506226,serie_6_1027_gao__port_506262,modus_1027_gao__port_506146,3_1027_gao__port_506113,405_1027_gao__port_506108,allroad_1027_gao__port_506297,auris_1027_gao__port_506322,galaxy_1027_gao__port_506143,giulietta_1027_gao__port_506363,106_1027_gao__port_506073,classe_m_1027_gao__port_506154,espace_1027_gao__port_506313,panda_1027_gao__port_506189,rcz_1027_gao__port_506197,4007_1027_gao__port_506162,classe_cl_1027_gao__port_506249,leaf_1027_gao__port_506139,octavia_1027_gao__port_506237,ds4_1027_gao__port_506336,freelander_1027_gao__port_506084,evasion_1027_gao__port_506109,punto_1027_gao__port_506106,2cv_1027_gao__port_506045,x4_1027_gao__port_506392,antara_1027_gao__port_506247,murano_1027_gao__port_506316,alto_1027_gao__port_506201,meriva_1027_gao__port_506353,orlando_1027_gao__port_506305,new_beetle_1027_gao__port_506050,306_1027_gao__port_506145,tiguan_1027_gao__port_506362,s_type_1027_gao__port_506101,c1_1027_gao__port_506128,vectra_1027_gao__port_506044,outlander_1027_gao__port_506317,307_1027_gao__port_506074,a6_s6_1027_gao__port_506134,nemo_combi_1027_gao__port_506196,berlingo_1027_gao__port_506194,partner_1027_gao__port_506285,cayenne_1027_gao__port_506177,quattroporte_1027_gao__port_506240,c_max_1027_gao__port_506282,fabia_1027_gao__port_506396,cx_3_1027_gao__port_506281,x_trail_1027_gao__port_506264,scirocco_1027_gao__port_506276,matiz_1027_gao__port_506144,tigra_1027_gao__port_506069,escort_1027_gao__port_506091,c2_1027_gao__port_506081,mini_1027_gao__port_506168,i30_1027_gao__port_506291,picanto_1027_gao__port_506238,mito_1027_gao__port_506072,impreza_1027_gao__port_506085,kangoo_1027_gao__port_506235,a4_1027_gao__port_506193,cayman_1027_gao__port_506268,sportage_1027_gao__port_506148,up_1027_gao__port_506356,optima_1027_gao__port_506386,defender_1027_gao__port_506229,serie_2_1027_gao__port_506256,edge_1027_gao__port_506187,r19_1027_gao__port_506110,jetta_1027_gao__port_506304,eos_1027_gao__port_506115,accord_1027_gao__port_506214,yaris_1027_gao__port_506334,classe_cls_1027_gao__port_506289,polo_1027_gao__port_506361,serie_4_1027_gao__port_506366,mini_cabriolet_1027_gao__port_506204,prius_1027_gao__port_506190,lodgy_1027_gao__port_506188,serie_7_1027_gao__port_506307,c15_1027_gao__port_506055,kadjar_1027_gao__port_506389,insignia_1027_gao__port_506364,308_1027_gao__port_506279,roomster_1027_gao__port_506241,80_1027_gao__port_506057,309_1027_gao__port_506063,tucson_1027_gao__port_506320,x3_1027_gao__port_506212,xf_1027_gao__port_506263,2008_1027_gao__port_506394,passat_1027_gao__port_506306,compass_1027_gao__port_506260,twingo_1027_gao__port_506309,micra_1027_gao__port_506221,golf_1027_gao__port_506155,soul_1027_gao__port_506176,rapid_1027_gao__port_506398,forester_1027_gao__port_506360,slk_1027_gao__port_506210,forfour_1027_gao__port_506341,serie_5_1027_gao__port_506209,xj_1027_gao__port_506170,pajero_1027_gao__port_506097,agila_1027_gao__port_506119,a6_1027_gao__port_506163,fox_1027_gao__port_506092,boxster_1027_gao__port_506267,altea_1027_gao__port_506246,samurai_1027_gao__port_506047,trax_1027_gao__port_506296,getz_1027_gao__port_506058,cherokee_1027_gao__port_506269,koleos_1027_gao__port_506378,z_series_1027_gao__port_506123,ecosport_1027_gao__port_506271,space_star_1027_gao__port_506277,rs3_sportback_1027_gao__port_506207,civic_1027_gao__port_506141,talisman_1027_gao__port_506390,f_pace_1027_gao__port_506314,classe_c_1027_gao__port_506299,tt_1027_gao__port_506075,pathfinder_1027_gao__port_506183,156_1027_gao__port_506157,cx_5_1027_gao__port_506228,scenic_1027_gao__port_506255,yeti_1027_gao__port_506358,mustang_1027_gao__port_506053,stilo_1027_gao__port_506060,ateca_1027_gao__port_506382,fiorino_1027_gao__port_506217,classe_glk_1027_gao__port_506290,fortwo_1027_gao__port_506230,cruze_1027_gao__port_506186,107_1027_gao__port_506213,aygo_1027_gao__port_506248,rx_1027_gao__port_506354,500_1027_gao__port_506245,bora_1027_gao__port_506104,transit_1027_gao__port_506111,pt_cruiser_1027_gao__port_506054,patrol_1027_gao__port_506068,r8_1027_gao__port_506156,xm_1027_gao__port_506102,s60_1027_gao__port_506191,aveo_1027_gao__port_506158,captiva_1027_gao__port_506159,ax_1027_gao__port_506153,rexton_1027_gao__port_506107,camaro_1027_gao__port_506056,ypsilon_1027_gao__port_506131,delta_1027_gao__port_506165,c4_1027_gao__port_506370,zx_1027_gao__port_506161,verso_1027_gao__port_506242,superb_1027_gao__port_506327,r5_1027_gao__port_506253,caddy_1027_gao__port_506330,x5_1027_gao__port_506243,f_type_1027_gao__port_506231,fusion_1027_gao__port_506096,dokker_1027_gao__port_506331,205_1027_gao__port_506062,macan_1027_gao__port_506195,tourneo_1027_gao__port_506369,108_1027_gao__port_506384,9_3_1027_gao__port_506071,mondeo_1027_gao__port_506116,cr_v_1027_gao__port_506164,c30_1027_gao__port_506090,pulsar_1027_gao__port_506397,ibiza_1027_gao__port_506273,a1_1027_gao__port_506338,matrix_1027_gao__port_506140,carnival_1027_gao__port_506136,xantia_1027_gao__port_506086,terrano_1027_gao__port_506083,q3_1027_gao__port_506275,hr_v_1027_gao__port_506283,expert_1027_gao__port_506142,multivan_1027_gao__port_506383,venga_1027_gao__port_506380,scudo_1027_gao__port_506129,laguna_1027_gao__port_506368,vel_satis_1027_gao__port_506130,b_max_1027_gao__port_506367,ignis_1027_gao__port_506292,159_1027_gao__port_506064,grande_punto_1027_gao__port_506138,logan_1027_gao__port_506167,s_max_1027_gao__port_506223,caravelle_1027_gao__port_506351,adam_1027_gao__port_506079,406_1027_gao__port_506132,q30_1027_gao__port_506293,almera_1027_gao__port_506089,corsa_1027_gao__port_506095,corolla_1027_gao__port_506120,xc60_1027_gao__port_506388,viano_1027_gao__port_506211,pro_cee_d_1027_gao__port_506274,a3_1027_gao__port_506321,v50_1027_gao__port_506150,voyager_1027_gao__port_506169,corvette_1027_gao__port_506049,rio_1027_gao__port_506379,jazz_1027_gao__port_506252,200_1027_gao__port_506112,tts_1027_gao__port_506199,zafira_1027_gao__port_506287,asx_1027_gao__port_506266,607_1027_gao__port_506118,207_1027_gao__port_506103,classe_s_1027_gao__port_506301,c6_1027_gao__port_506105,express_1027_gao__port_506137,classe_gla_1027_gao__port_506352,v60_1027_gao__port_506333,ka_1027_gao__port_506180,range_rover_1027_gao__port_506254,discovery_1027_gao__port_506375,classe_r_1027_gao__port_506270,transporter_1027_gao__port_506319,cee_d_1027_gao__port_506288,zoe_1027_gao__port_506244,i20_1027_gao__port_506284,gtv_1027_gao__port_506059,s4_avant_1027_gao__port_506261,x1_1027_gao__port_506372,autres_1027_gao__port_506127,208_1027_gao__port_506359,c8_1027_gao__port_506135,astra_1027_gao__port_506215,2_1027_gao__port_506151,doblo_1027_gao__port_506251,807_1027_gao__port_506152,206_1027_gao__port_506126,a7_1027_gao__port_506373,renegade_1027_gao__port_506346', 'svm_portfolios_learning': '506302,506374,506399,506192,506205,506350,506052,506295,506066,506117,506065,506125,506387,506381,506349,506328,506377,506286,506124,506172,506206,506178,506371,506076,506114,506329,506122,506220,506174,506224,506232,506234,506173,506181,506323,506326,506376,506048,506400,506179,506311,506325,506402,506051,506294,506318,506303,506175,506099,506061,506337,506250,506082,506166,506133,506308,506078,506340,506310,506100,506121,506070,506218,506227,506272,506147,506160,506265,506202,506222,506093,506257,506208,506344,506077,506395,506094,506219,506298,506339,506343,506365,506200,506348,506198,506385,506239,506236,506391,506087,506342,506149,506184,506393,506203,506280,506216,506403,506355,506332,506259,506401,506357,506324,506098,506315,506335,506088,506046,506185,506171,506080,506345,506347,506067,506233,506225,506312,506278,506300,506258,506182,506226,506262,506146,506113,506108,506297,506322,506143,506363,506073,506154,506313,506189,506197,506162,506249,506139,506237,506336,506084,506109,506106,506045,506392,506247,506316,506201,506353,506305,506050,506145,506362,506101,506128,506044,506317,506074,506134,506196,506194,506285,506177,506240,506282,506396,506281,506264,506276,506144,506069,506091,506081,506168,506291,506238,506072,506085,506235,506193,506268,506148,506356,506386,506229,506256,506187,506110,506304,506115,506214,506334,506289,506361,506366,506204,506190,506188,506307,506055,506389,506364,506279,506241,506057,506063,506320,506212,506263,506394,506306,506260,506309,506221,506155,506176,506398,506360,506210,506341,506209,506170,506097,506119,506163,506092,506267,506246,506047,506296,506058,506269,506378,506123,506271,506277,506207,506141,506390,506314,506299,506075,506183,506157,506228,506255,506358,506053,506060,506382,506217,506290,506230,506186,506213,506248,506354,506245,506104,506111,506054,506068,506156,506102,506191,506158,506159,506153,506107,506056,506131,506165,506370,506161,506242,506327,506253,506330,506243,506231,506096,506331,506062,506195,506369,506384,506071,506116,506164,506090,506397,506273,506338,506140,506136,506086,506083,506275,506283,506142,506383,506380,506129,506368,506130,506367,506292,506064,506138,506167,506223,506351,506079,506132,506293,506089,506095,506120,506388,506211,506274,506321,506150,506169,506049,506379,506252,506112,506199,506287,506266,506118,506103,506301,506105,506137,506352,506333,506180,506254,506375,506270,506319,506288,506244,506284,506059,506261,506372,506127,506359,506135,506215,506151,506251,506152,506126,506373,506346', 'photo_hashtag_type': 332, 'photo_desc_type': 3390, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3390 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (3390) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3390, 'car_360_1027', 16384, 25088, 'car_360_1027', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2017, 10, 28, 12, 29, 27), datetime.datetime(2017, 10, 28, 12, 29, 27)) To loadFromThcl() : net_3390 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 1428 wait 20 seconds l 3637 free memory gpu now : 1428 max_wait_temp : 1 max_wait : 0 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (3390) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3390, 'car_360_1027', 16384, 25088, 'car_360_1027', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2017, 10, 28, 12, 29, 27), datetime.datetime(2017, 10, 28, 12, 29, 27)) param : , param.caffemodel : car_360_1027 None mean_file_type : mean_file_path : prototxt_file_path : model : car_360_1027 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : car_360_1027 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : [] local folder : /data/models_weight/car_360_1027 /data/models_weight/car_360_1027/caffemodel size_local : 542944640 size in s3 : 542944640 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:43 caffemodel already exist and didn't need to update /data/models_weight/car_360_1027/deploy_conv_normal.prototxt size_local : 4626 size in s3 : 4626 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:42 deploy_conv_normal.prototxt already exist and didn't need to update /data/models_weight/car_360_1027/deploy_fc.prototxt size_local : 1132 size in s3 : 1132 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:43 deploy_fc.prototxt already exist and didn't need to update /data/models_weight/car_360_1027/deploy.prototxt size_local : 5654 size in s3 : 5654 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:42 deploy.prototxt already exist and didn't need to update /data/models_weight/car_360_1027/mean.npy size_local : 1572944 size in s3 : 1572944 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:55 mean.npy already exist and didn't need to update /data/models_weight/car_360_1027/synset_words.txt size_local : 13687 size in s3 : 13687 create time local : 2021-08-09 05:28:34 create time in s3 : 2021-08-06 17:57:43 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/car_360_1027/deploy.prototxt caffemodel_filename : /data/models_weight/car_360_1027/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 1428 wait 20 seconds l 3637 free memory gpu now : 1428 max_wait_temp : 1 max_wait : 0 dict_keys(['prob', 'pool5']) time used to do the prepocess of the images : 0.07042670249938965 time used to do the prediction : 0.18683576583862305 save descriptor for thcl : 355 (1, 512, 7, 7) Got the blobs of the net to insert : [0, 0, 0, 0, 0, 0, 0, 0, 0, 0] code_as_byte_string:b'0000000000'| time to traite the descriptors : 0.050498008728027344 Testing : ['916235064'] In select_photos_meta_from_ids: SELECT photo_id, url, FROM_UNIXTIME(uploaded_at), latitude, longitude, text FROM MTRBack.photos WHERE photo_id IN (916235064) result : {916235064: {'photo_id': 916235064, 'url': 'https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/2017/10/14/6293d1bb790dc6902450e7c572b7d10b.jpg', 'latitude': 0.0, 'longitude': 0.0, 'text': None}} list_photo_exists : [916235064] storage_type for insertDescriptorsMulti : 1 To insert : 916235064 time to insert the descriptors : 0.8068792819976807 After datou_step_exec type output : map_portfolio_photo : len 0 keys : dict_keys([]) Inside saveOutput : final : False verbose : True time used to find the portfolios of the photos select button_legend_list from MTRDatou.classification_theme where id = 355 SAVE THCL, output : {'916235064': [[('916235064', 'c_elysee_1027_gao__port_506302', 0.001881405, 332, '355'), ('916235064', 'mokka_1027_gao__port_506374', 0.0011635128, 332, '355'), ('916235064', 'captur_1027_gao__port_506399', 0.0008158046, 332, '355'), ('916235064', 'sorento_1027_gao__port_506192', 0.001177459, 332, '355'), ('916235064', 'navara_1027_gao__port_506205', 0.0025848972, 332, '355'), ('916235064', 'xc90_1027_gao__port_506350', 0.004170162, 332, '355'), ('916235064', 'saxo_1027_gao__port_506052', 0.003480807, 332, '355'), ('916235064', 'trafic_1027_gao__port_506295', 0.0073649185, 332, '355'), ('916235064', 'punto_evo_1027_gao__port_506066', 0.0021888916, 332, '355'), ('916235064', '5_1027_gao__port_506117', 0.00057982473, 332, '355'), ('916235064', '250_1027_gao__port_506065', 0.0045921276, 332, '355'), ('916235064', 'd_max_1027_gao__port_506125', 0.0031587605, 332, '355'), ('916235064', 'panamera_1027_gao__port_506387', 0.0022508202, 332, '355'), ('916235064', 'alhambra_1027_gao__port_506381', 0.0053196982, 332, '355'), ('916235064', 'x6_1027_gao__port_506349', 0.0011002271, 332, '355'), ('916235064', 'vitara_1027_gao__port_506328', 0.0054021566, 332, '355'), ('916235064', 'fiesta_1027_gao__port_506377', 0.00391906, 332, '355'), ('916235064', 'qashqai_1027_gao__port_506286', 0.0014786993, 332, '355'), ('916235064', '147_1027_gao__port_506124', 0.0019779543, 332, '355'), ('916235064', 'c5_1027_gao__port_506172', 0.0012443673, 332, '355'), ('916235064', 'q5_1027_gao__port_506206', 0.0015050521, 332, '355'), ('916235064', 'giulia_1027_gao__port_506178', 0.0021695797, 332, '355'), ('916235064', 'karl_1027_gao__port_506371', 0.0027072902, 332, '355'), ('916235064', 'mehari_1027_gao__port_506076', 0.0047054263, 332, '355'), ('916235064', '911_1027_gao__port_506114', 0.001942278, 332, '355'), ('916235064', '508_1027_gao__port_506329', 0.0009585849, 332, '355'), ('916235064', 'idea_1027_gao__port_506122', 0.00077005307, 332, '355'), ('916235064', 'megane_1027_gao__port_506220', 0.0019470484, 332, '355'), ('916235064', 'ghibli_1027_gao__port_506174', 0.0013723855, 332, '355'), ('916235064', 'touareg_1027_gao__port_506224', 0.0016204519, 332, '355'), ('916235064', 'i10_1027_gao__port_506232', 0.0013922948, 332, '355'), ('916235064', 'jumper_1027_gao__port_506234', 0.01004077, 332, '355'), ('916235064', 'classe_clk_1027_gao__port_506173', 0.0010793965, 332, '355'), ('916235064', 'kuga_1027_gao__port_506181', 0.0008446531, 332, '355'), ('916235064', 'ct_1027_gao__port_506323', 0.0012521491, 332, '355'), ('916235064', 'leon_1027_gao__port_506326', 0.0025843773, 332, '355'), ('916235064', 'ds5_1027_gao__port_506376', 0.0012431119, 332, '355'), ('916235064', 'cordoba_1027_gao__port_506048', 0.0028651964, 332, '355'), ('916235064', 'classe_cla_1027_gao__port_506400', 0.0012948806, 332, '355'), ('916235064', 'jumpy_1027_gao__port_506179', 0.0103363, 332, '355'), ('916235064', 'avensis_1027_gao__port_506311', 0.0018770705, 332, '355'), ('916235064', 'juke_1027_gao__port_506325', 0.0011343489, 332, '355'), ('916235064', '4008_1027_gao__port_506402', 0.0015758214, 332, '355'), ('916235064', '190_series_1027_gao__port_506051', 0.0039823446, 332, '355'), ('916235064', 'serie_3_1027_gao__port_506294', 0.0028745728, 332, '355'), ('916235064', 'q7_1027_gao__port_506318', 0.0023355382, 332, '355'), ('916235064', 'glc_1027_gao__port_506303', 0.0012105894, 332, '355'), ('916235064', 'grand_vitara_1027_gao__port_506175', 0.001144665, 332, '355'), ('916235064', 's40_1027_gao__port_506099', 0.0022341053, 332, '355'), ('916235064', 'toledo_1027_gao__port_506061', 0.001746329, 332, '355'), ('916235064', '5008_1027_gao__port_506337', 0.0046986225, 332, '355'), ('916235064', 'continental_1027_gao__port_506250', 0.002191817, 332, '355'), ('916235064', 'coupe_1027_gao__port_506082', 0.002264014, 332, '355'), ('916235064', 'iq_1027_gao__port_506166', 0.0018171524, 332, '355'), ('916235064', '407_1027_gao__port_506133', 0.00090580113, 332, '355'), ('916235064', 'touran_1027_gao__port_506308', 0.00203998, 332, '355'), ('916235064', '300c_1027_gao__port_506078', 0.0025340111, 332, '355'), ('916235064', 'classe_gl_1027_gao__port_506340', 0.0044886884, 332, '355'), ('916235064', 'vivaro_1027_gao__port_506310', 0.003424752, 332, '355'), ('916235064', 'sl_1027_gao__port_506100', 0.0031362008, 332, '355'), ('916235064', 'elise_1027_gao__port_506121', 0.0010257903, 332, '355'), ('916235064', '1007_1027_gao__port_506070', 0.0015351687, 332, '355'), ('916235064', 'i40_1027_gao__port_506218', 0.00059153285, 332, '355'), ('916235064', 'bipper_tepee_1027_gao__port_506227', 0.004027876, 332, '355'), ('916235064', 'focus_1027_gao__port_506272', 0.0011587115, 332, '355'), ('916235064', 'primera_1027_gao__port_506147', 0.0012159357, 332, '355'), ('916235064', 'r4_1027_gao__port_506160', 0.014971194, 332, '355'), ('916235064', 'a8_1027_gao__port_506265', 0.0011323855, 332, '355'), ('916235064', 'boxer_1027_gao__port_506202', 0.010542169, 332, '355'), ('916235064', 's5_1027_gao__port_506222', 0.0011987712, 332, '355'), ('916235064', 'r21_1027_gao__port_506093', 0.004187547, 332, '355'), ('916235064', 'c3_1027_gao__port_506257', 0.0023633665, 332, '355'), ('916235064', 'santa_fe_1027_gao__port_506208', 0.0016323832, 332, '355'), ('916235064', 'm4_1027_gao__port_506344', 0.0015569492, 332, '355'), ('916235064', 'safrane_1027_gao__port_506077', 0.0013962379, 332, '355'), ('916235064', 'classe_gle_1027_gao__port_506395', 0.002198199, 332, '355'), ('916235064', '0_1027_gao__port_506094', 0.008829112, 332, '355'), ('916235064', 'ix35_1027_gao__port_506219', 0.0014614249, 332, '355'), ('916235064', 'carens_1027_gao__port_506298', 0.0008825688, 332, '355'), ('916235064', 'classe_a_1027_gao__port_506339', 0.0024709995, 332, '355'), ('916235064', 'ix20_1027_gao__port_506343', 0.0010091326, 332, '355'), ('916235064', 'note_1027_gao__port_506365', 0.00159595, 332, '355'), ('916235064', 'a5_1027_gao__port_506200', 0.0015331754, 332, '355'), ('916235064', 'sx4_1027_gao__port_506348', 0.0014914296, 332, '355'), ('916235064', 'sandero_1027_gao__port_506198', 0.0014584624, 332, '355'), ('916235064', '3008_1027_gao__port_506385', 0.0056449207, 332, '355'), ('916235064', 'q50_1027_gao__port_506239', 0.0011166334, 332, '355'), ('916235064', 'latitude_1027_gao__port_506236', 0.0008020163, 332, '355'), ('916235064', 'v40_1027_gao__port_506391', 0.0017145631, 332, '355'), ('916235064', 'xsara_1027_gao__port_506087', 0.0009824896, 332, '355'), ('916235064', 'grand_c_max_1027_gao__port_506342', 0.0017957059, 332, '355'), ('916235064', 'swift_1027_gao__port_506149', 0.0015019191, 332, '355'), ('916235064', 'serie_1_1027_gao__port_506184', 0.0015141121, 332, '355'), ('916235064', 'xc70_1027_gao__port_506393', 0.0036189996, 332, '355'), ('916235064', 'master_1027_gao__port_506203', 0.007955822, 332, '355'), ('916235064', 'clio_1027_gao__port_506280', 0.0029573974, 332, '355'), ('916235064', 'duster_1027_gao__port_506216', 0.00074429886, 332, '355'), ('916235064', 'traveller_1027_gao__port_506403', 0.0042925654, 332, '355'), ('916235064', 'tipo_1027_gao__port_506355', 0.0010930217, 332, '355'), ('916235064', 'rav_4_1027_gao__port_506332', 0.001360405, 332, '355'), ('916235064', 'coccinelle_1027_gao__port_506259', 0.003494882, 332, '355'), ('916235064', 'spacetourer_1027_gao__port_506401', 0.0030964876, 332, '355'), ('916235064', 'xe_1027_gao__port_506357', 0.0014474166, 332, '355'), ('916235064', 'ds3_1027_gao__port_506324', 0.0013094675, 332, '355'), ('916235064', 'mx_5_1027_gao__port_506098', 0.0025890768, 332, '355'), ('916235064', 'land_cruiser_1027_gao__port_506315', 0.00953173, 332, '355'), ('916235064', 'classe_b_1027_gao__port_506335', 0.0017214247, 332, '355'), ('916235064', '806_1027_gao__port_506088', 0.0025620277, 332, '355'), ('916235064', 'rx_8_1027_gao__port_506046', 0.0036222541, 332, '355'), ('916235064', 'spark_1027_gao__port_506185', 0.0010075802, 332, '355'), ('916235064', '6_1027_gao__port_506171', 0.0011183529, 332, '355'), ('916235064', 'bravo_1027_gao__port_506080', 0.0014650803, 332, '355'), ('916235064', 'nx_1027_gao__port_506345', 0.0013684954, 332, '355'), ('916235064', 'sharan_1027_gao__port_506347', 0.0050913305, 332, '355'), ('916235064', 'x_type_1027_gao__port_506067', 0.0007802942, 332, '355'), ('916235064', 'jimny_1027_gao__port_506233', 0.0046052793, 332, '355'), ('916235064', 'wrangler_1027_gao__port_506225', 0.001799918, 332, '355'), ('916235064', 'c_crosser_1027_gao__port_506312', 0.0015926941, 332, '355'), ('916235064', 'v70_1027_gao__port_506278', 0.0019680476, 332, '355'), ('916235064', 'classe_e_1027_gao__port_506300', 0.0017371367, 332, '355'), ('916235064', 'classe_v_1027_gao__port_506258', 0.0127258245, 332, '355'), ('916235064', 'm3_1027_gao__port_506182', 0.002338047, 332, '355'), ('916235064', 'abarth_500_1027_gao__port_506226', 0.0040436485, 332, '355'), ('916235064', 'serie_6_1027_gao__port_506262', 0.0011316322, 332, '355'), ('916235064', 'modus_1027_gao__port_506146', 0.0018290002, 332, '355'), ('916235064', '3_1027_gao__port_506113', 0.0015081372, 332, '355'), ('916235064', '405_1027_gao__port_506108', 0.008056917, 332, '355'), ('916235064', 'allroad_1027_gao__port_506297', 0.0010598884, 332, '355'), ('916235064', 'auris_1027_gao__port_506322', 0.001152486, 332, '355'), ('916235064', 'galaxy_1027_gao__port_506143', 0.002514918, 332, '355'), ('916235064', 'giulietta_1027_gao__port_506363', 0.0008652153, 332, '355'), ('916235064', '106_1027_gao__port_506073', 0.008270163, 332, '355'), ('916235064', 'classe_m_1027_gao__port_506154', 0.0030016904, 332, '355'), ('916235064', 'espace_1027_gao__port_506313', 0.0010645459, 332, '355'), ('916235064', 'panda_1027_gao__port_506189', 0.009028754, 332, '355'), ('916235064', 'rcz_1027_gao__port_506197', 0.001129447, 332, '355'), ('916235064', '4007_1027_gao__port_506162', 0.0006792379, 332, '355'), ('916235064', 'classe_cl_1027_gao__port_506249', 0.00108627, 332, '355'), ('916235064', 'leaf_1027_gao__port_506139', 0.0018037749, 332, '355'), ('916235064', 'octavia_1027_gao__port_506237', 0.001860112, 332, '355'), ('916235064', 'ds4_1027_gao__port_506336', 0.0024159981, 332, '355'), ('916235064', 'freelander_1027_gao__port_506084', 0.0023476458, 332, '355'), ('916235064', 'evasion_1027_gao__port_506109', 0.0031142824, 332, '355'), ('916235064', 'punto_1027_gao__port_506106', 0.0019496459, 332, '355'), ('916235064', '2cv_1027_gao__port_506045', 0.0079763625, 332, '355'), ('916235064', 'x4_1027_gao__port_506392', 0.0017949659, 332, '355'), ('916235064', 'antara_1027_gao__port_506247', 0.0012469388, 332, '355'), ('916235064', 'murano_1027_gao__port_506316', 0.00060894416, 332, '355'), ('916235064', 'alto_1027_gao__port_506201', 0.009229263, 332, '355'), ('916235064', 'meriva_1027_gao__port_506353', 0.0013762492, 332, '355'), ('916235064', 'orlando_1027_gao__port_506305', 0.0018461219, 332, '355'), ('916235064', 'new_beetle_1027_gao__port_506050', 0.0011639573, 332, '355'), ('916235064', '306_1027_gao__port_506145', 0.0035072332, 332, '355'), ('916235064', 'tiguan_1027_gao__port_506362', 0.0026823322, 332, '355'), ('916235064', 's_type_1027_gao__port_506101', 0.0011383992, 332, '355'), ('916235064', 'c1_1027_gao__port_506128', 0.0027512347, 332, '355'), ('916235064', 'vectra_1027_gao__port_506044', 0.0011992564, 332, '355'), ('916235064', 'outlander_1027_gao__port_506317', 0.0017119196, 332, '355'), ('916235064', '307_1027_gao__port_506074', 0.0020011528, 332, '355'), ('916235064', 'a6_s6_1027_gao__port_506134', 0.0016576415, 332, '355'), ('916235064', 'nemo_combi_1027_gao__port_506196', 0.0022653628, 332, '355'), ('916235064', 'berlingo_1027_gao__port_506194', 0.0046610488, 332, '355'), ('916235064', 'partner_1027_gao__port_506285', 0.0039404975, 332, '355'), ('916235064', 'cayenne_1027_gao__port_506177', 0.003797788, 332, '355'), ('916235064', 'quattroporte_1027_gao__port_506240', 0.0024443741, 332, '355'), ('916235064', 'c_max_1027_gao__port_506282', 0.0013124063, 332, '355'), ('916235064', 'fabia_1027_gao__port_506396', 0.0052992, 332, '355'), ('916235064', 'cx_3_1027_gao__port_506281', 0.0014462165, 332, '355'), ('916235064', 'x_trail_1027_gao__port_506264', 0.0018315469, 332, '355'), ('916235064', 'scirocco_1027_gao__port_506276', 0.004790846, 332, '355'), ('916235064', 'matiz_1027_gao__port_506144', 0.0017561303, 332, '355'), ('916235064', 'tigra_1027_gao__port_506069', 0.0008544126, 332, '355'), ('916235064', 'escort_1027_gao__port_506091', 0.0048417407, 332, '355'), ('916235064', 'c2_1027_gao__port_506081', 0.0014904465, 332, '355'), ('916235064', 'mini_1027_gao__port_506168', 0.0011924049, 332, '355'), ('916235064', 'i30_1027_gao__port_506291', 0.0006325375, 332, '355'), ('916235064', 'picanto_1027_gao__port_506238', 0.0029746504, 332, '355'), ('916235064', 'mito_1027_gao__port_506072', 0.0015077343, 332, '355'), ('916235064', 'impreza_1027_gao__port_506085', 0.002019473, 332, '355'), ('916235064', 'kangoo_1027_gao__port_506235', 0.006582703, 332, '355'), ('916235064', 'a4_1027_gao__port_506193', 0.001987646, 332, '355'), ('916235064', 'cayman_1027_gao__port_506268', 0.0018143053, 332, '355'), ('916235064', 'sportage_1027_gao__port_506148', 0.0014274905, 332, '355'), ('916235064', 'up_1027_gao__port_506356', 0.006861969, 332, '355'), ('916235064', 'optima_1027_gao__port_506386', 0.0008919986, 332, '355'), ('916235064', 'defender_1027_gao__port_506229', 0.006722292, 332, '355'), ('916235064', 'serie_2_1027_gao__port_506256', 0.0022276593, 332, '355'), ('916235064', 'edge_1027_gao__port_506187', 0.000874874, 332, '355'), ('916235064', 'r19_1027_gao__port_506110', 0.0049436954, 332, '355'), ('916235064', 'jetta_1027_gao__port_506304', 0.0036194327, 332, '355'), ('916235064', 'eos_1027_gao__port_506115', 0.0038927698, 332, '355'), ('916235064', 'accord_1027_gao__port_506214', 0.002013419, 332, '355'), ('916235064', 'yaris_1027_gao__port_506334', 0.0032331496, 332, '355'), ('916235064', 'classe_cls_1027_gao__port_506289', 0.00078528124, 332, '355'), ('916235064', 'polo_1027_gao__port_506361', 0.0043101306, 332, '355'), ('916235064', 'serie_4_1027_gao__port_506366', 0.0011477651, 332, '355'), ('916235064', 'mini_cabriolet_1027_gao__port_506204', 0.0008379063, 332, '355'), ('916235064', 'prius_1027_gao__port_506190', 0.001149024, 332, '355'), ('916235064', 'lodgy_1027_gao__port_506188', 0.0020171886, 332, '355'), ('916235064', 'serie_7_1027_gao__port_506307', 0.0012481364, 332, '355'), ('916235064', 'c15_1027_gao__port_506055', 0.017712602, 332, '355'), ('916235064', 'kadjar_1027_gao__port_506389', 0.0012505422, 332, '355'), ('916235064', 'insignia_1027_gao__port_506364', 0.0016433424, 332, '355'), ('916235064', '308_1027_gao__port_506279', 0.002123317, 332, '355'), ('916235064', 'roomster_1027_gao__port_506241', 0.0018010826, 332, '355'), ('916235064', '80_1027_gao__port_506057', 0.0046150233, 332, '355'), ('916235064', '309_1027_gao__port_506063', 0.013525933, 332, '355'), ('916235064', 'tucson_1027_gao__port_506320', 0.002123598, 332, '355'), ('916235064', 'x3_1027_gao__port_506212', 0.00089779386, 332, '355'), ('916235064', 'xf_1027_gao__port_506263', 0.0011167723, 332, '355'), ('916235064', '2008_1027_gao__port_506394', 0.0026408557, 332, '355'), ('916235064', 'passat_1027_gao__port_506306', 0.0014978182, 332, '355'), ('916235064', 'compass_1027_gao__port_506260', 0.0032566036, 332, '355'), ('916235064', 'twingo_1027_gao__port_506309', 0.00649759, 332, '355'), ('916235064', 'micra_1027_gao__port_506221', 0.0035854836, 332, '355'), ('916235064', 'golf_1027_gao__port_506155', 0.0031929952, 332, '355'), ('916235064', 'soul_1027_gao__port_506176', 0.0012889907, 332, '355'), ('916235064', 'rapid_1027_gao__port_506398', 0.00259136, 332, '355'), ('916235064', 'forester_1027_gao__port_506360', 0.00227641, 332, '355'), ('916235064', 'slk_1027_gao__port_506210', 0.0015846448, 332, '355'), ('916235064', 'forfour_1027_gao__port_506341', 0.0021794525, 332, '355'), ('916235064', 'serie_5_1027_gao__port_506209', 0.0013697561, 332, '355'), ('916235064', 'xj_1027_gao__port_506170', 0.0026014866, 332, '355'), ('916235064', 'pajero_1027_gao__port_506097', 0.005211631, 332, '355'), ('916235064', 'agila_1027_gao__port_506119', 0.004849533, 332, '355'), ('916235064', 'a6_1027_gao__port_506163', 0.0018915972, 332, '355'), ('916235064', 'fox_1027_gao__port_506092', 0.00084662734, 332, '355'), ('916235064', 'boxster_1027_gao__port_506267', 0.0015943744, 332, '355'), ('916235064', 'altea_1027_gao__port_506246', 0.0021389609, 332, '355'), ('916235064', 'samurai_1027_gao__port_506047', 0.006253163, 332, '355'), ('916235064', 'trax_1027_gao__port_506296', 0.0019437515, 332, '355'), ('916235064', 'getz_1027_gao__port_506058', 0.0016386389, 332, '355'), ('916235064', 'cherokee_1027_gao__port_506269', 0.0029814325, 332, '355'), ('916235064', 'koleos_1027_gao__port_506378', 0.0015592134, 332, '355'), ('916235064', 'z_series_1027_gao__port_506123', 0.0016566475, 332, '355'), ('916235064', 'ecosport_1027_gao__port_506271', 0.0013229565, 332, '355'), ('916235064', 'space_star_1027_gao__port_506277', 0.002114046, 332, '355'), ('916235064', 'rs3_sportback_1027_gao__port_506207', 0.0019119023, 332, '355'), ('916235064', 'civic_1027_gao__port_506141', 0.0026903006, 332, '355'), ('916235064', 'talisman_1027_gao__port_506390', 0.00076135556, 332, '355'), ('916235064', 'f_pace_1027_gao__port_506314', 0.0016166983, 332, '355'), ('916235064', 'classe_c_1027_gao__port_506299', 0.0017943591, 332, '355'), ('916235064', 'tt_1027_gao__port_506075', 0.0013937561, 332, '355'), ('916235064', 'pathfinder_1027_gao__port_506183', 0.0016512058, 332, '355'), ('916235064', '156_1027_gao__port_506157', 0.0015444808, 332, '355'), ('916235064', 'cx_5_1027_gao__port_506228', 0.0014414112, 332, '355'), ('916235064', 'scenic_1027_gao__port_506255', 0.0016085037, 332, '355'), ('916235064', 'yeti_1027_gao__port_506358', 0.002091073, 332, '355'), ('916235064', 'mustang_1027_gao__port_506053', 0.0100543015, 332, '355'), ('916235064', 'stilo_1027_gao__port_506060', 0.0010832578, 332, '355'), ('916235064', 'ateca_1027_gao__port_506382', 0.0017012329, 332, '355'), ('916235064', 'fiorino_1027_gao__port_506217', 0.009196709, 332, '355'), ('916235064', 'classe_glk_1027_gao__port_506290', 0.0017016606, 332, '355'), ('916235064', 'fortwo_1027_gao__port_506230', 0.0016009267, 332, '355'), ('916235064', 'cruze_1027_gao__port_506186', 0.001005357, 332, '355'), ('916235064', '107_1027_gao__port_506213', 0.0016274743, 332, '355'), ('916235064', 'aygo_1027_gao__port_506248', 0.0032430983, 332, '355'), ('916235064', 'rx_1027_gao__port_506354', 0.0010633438, 332, '355'), ('916235064', '500_1027_gao__port_506245', 0.0016355662, 332, '355'), ('916235064', 'bora_1027_gao__port_506104', 0.003816246, 332, '355'), ('916235064', 'transit_1027_gao__port_506111', 0.004861494, 332, '355'), ('916235064', 'pt_cruiser_1027_gao__port_506054', 0.0019164234, 332, '355'), ('916235064', 'patrol_1027_gao__port_506068', 0.0042409385, 332, '355'), ('916235064', 'r8_1027_gao__port_506156', 0.001272444, 332, '355'), ('916235064', 'xm_1027_gao__port_506102', 0.0022684815, 332, '355'), ('916235064', 's60_1027_gao__port_506191', 0.003199291, 332, '355'), ('916235064', 'aveo_1027_gao__port_506158', 0.0038393429, 332, '355'), ('916235064', 'captiva_1027_gao__port_506159', 0.0017188187, 332, '355'), ('916235064', 'ax_1027_gao__port_506153', 0.006898715, 332, '355'), ('916235064', 'rexton_1027_gao__port_506107', 0.0013022242, 332, '355'), ('916235064', 'camaro_1027_gao__port_506056', 0.002490989, 332, '355'), ('916235064', 'ypsilon_1027_gao__port_506131', 0.0019542356, 332, '355'), ('916235064', 'delta_1027_gao__port_506165', 0.0014005479, 332, '355'), ('916235064', 'c4_1027_gao__port_506370', 0.0013009832, 332, '355'), ('916235064', 'zx_1027_gao__port_506161', 0.004594893, 332, '355'), ('916235064', 'verso_1027_gao__port_506242', 0.0007721042, 332, '355'), ('916235064', 'superb_1027_gao__port_506327', 0.0019943707, 332, '355'), ('916235064', 'r5_1027_gao__port_506253', 0.009547346, 332, '355'), ('916235064', 'caddy_1027_gao__port_506330', 0.013826109, 332, '355'), ('916235064', 'x5_1027_gao__port_506243', 0.0011205694, 332, '355'), ('916235064', 'f_type_1027_gao__port_506231', 0.0008300661, 332, '355'), ('916235064', 'fusion_1027_gao__port_506096', 0.0012668984, 332, '355'), ('916235064', 'dokker_1027_gao__port_506331', 0.005355464, 332, '355'), ('916235064', '205_1027_gao__port_506062', 0.006685535, 332, '355'), ('916235064', 'macan_1027_gao__port_506195', 0.0015573286, 332, '355'), ('916235064', 'tourneo_1027_gao__port_506369', 0.0063991756, 332, '355'), ('916235064', '108_1027_gao__port_506384', 0.0052634743, 332, '355'), ('916235064', '9_3_1027_gao__port_506071', 0.0008374995, 332, '355'), ('916235064', 'mondeo_1027_gao__port_506116', 0.0014397606, 332, '355'), ('916235064', 'cr_v_1027_gao__port_506164', 0.0016412436, 332, '355'), ('916235064', 'c30_1027_gao__port_506090', 0.0017487968, 332, '355'), ('916235064', 'pulsar_1027_gao__port_506397', 0.0012017612, 332, '355'), ('916235064', 'ibiza_1027_gao__port_506273', 0.0037232332, 332, '355'), ('916235064', 'a1_1027_gao__port_506338', 0.0012347312, 332, '355'), ('916235064', 'matrix_1027_gao__port_506140', 0.00070769945, 332, '355'), ('916235064', 'carnival_1027_gao__port_506136', 0.0022811112, 332, '355'), ('916235064', 'xantia_1027_gao__port_506086', 0.0021964514, 332, '355'), ('916235064', 'terrano_1027_gao__port_506083', 0.002029553, 332, '355'), ('916235064', 'q3_1027_gao__port_506275', 0.0011265237, 332, '355'), ('916235064', 'hr_v_1027_gao__port_506283', 0.001780641, 332, '355'), ('916235064', 'expert_1027_gao__port_506142', 0.007366908, 332, '355'), ('916235064', 'multivan_1027_gao__port_506383', 0.006503836, 332, '355'), ('916235064', 'venga_1027_gao__port_506380', 0.0008004776, 332, '355'), ('916235064', 'scudo_1027_gao__port_506129', 0.0055911005, 332, '355'), ('916235064', 'laguna_1027_gao__port_506368', 0.00071348064, 332, '355'), ('916235064', 'vel_satis_1027_gao__port_506130', 0.0027266801, 332, '355'), ('916235064', 'b_max_1027_gao__port_506367', 0.0017245755, 332, '355'), ('916235064', 'ignis_1027_gao__port_506292', 0.0043554395, 332, '355'), ('916235064', '159_1027_gao__port_506064', 0.0010783031, 332, '355'), ('916235064', 'grande_punto_1027_gao__port_506138', 0.002363357, 332, '355'), ('916235064', 'logan_1027_gao__port_506167', 0.004396309, 332, '355'), ('916235064', 's_max_1027_gao__port_506223', 0.0012525861, 332, '355'), ('916235064', 'caravelle_1027_gao__port_506351', 0.0030292473, 332, '355'), ('916235064', 'adam_1027_gao__port_506079', 0.0010538646, 332, '355'), ('916235064', '406_1027_gao__port_506132', 0.0013577469, 332, '355'), ('916235064', 'q30_1027_gao__port_506293', 0.00097165967, 332, '355'), ('916235064', 'almera_1027_gao__port_506089', 0.0010239701, 332, '355'), ('916235064', 'corsa_1027_gao__port_506095', 0.00252038, 332, '355'), ('916235064', 'corolla_1027_gao__port_506120', 0.0026819515, 332, '355'), ('916235064', 'xc60_1027_gao__port_506388', 0.001898476, 332, '355'), ('916235064', 'viano_1027_gao__port_506211', 0.0026942298, 332, '355'), ('916235064', 'pro_cee_d_1027_gao__port_506274', 0.0008320816, 332, '355'), ('916235064', 'a3_1027_gao__port_506321', 0.0037384913, 332, '355'), ('916235064', 'v50_1027_gao__port_506150', 0.0007920721, 332, '355'), ('916235064', 'voyager_1027_gao__port_506169', 0.0030524528, 332, '355'), ('916235064', 'corvette_1027_gao__port_506049', 0.003723897, 332, '355'), ('916235064', 'rio_1027_gao__port_506379', 0.0017739582, 332, '355'), ('916235064', 'jazz_1027_gao__port_506252', 0.0015301626, 332, '355'), ('916235064', '200_1027_gao__port_506112', 0.0040881, 332, '355'), ('916235064', 'tts_1027_gao__port_506199', 0.0011864705, 332, '355'), ('916235064', 'zafira_1027_gao__port_506287', 0.0026955458, 332, '355'), ('916235064', 'asx_1027_gao__port_506266', 0.0011407478, 332, '355'), ('916235064', '607_1027_gao__port_506118', 0.0012530201, 332, '355'), ('916235064', '207_1027_gao__port_506103', 0.0015146541, 332, '355'), ('916235064', 'classe_s_1027_gao__port_506301', 0.0031662649, 332, '355'), ('916235064', 'c6_1027_gao__port_506105', 0.0017352523, 332, '355'), ('916235064', 'express_1027_gao__port_506137', 0.01672694, 332, '355'), ('916235064', 'classe_gla_1027_gao__port_506352', 0.0018254172, 332, '355'), ('916235064', 'v60_1027_gao__port_506333', 0.002145741, 332, '355'), ('916235064', 'ka_1027_gao__port_506180', 0.0014152287, 332, '355'), ('916235064', 'range_rover_1027_gao__port_506254', 0.0020554322, 332, '355'), ('916235064', 'discovery_1027_gao__port_506375', 0.002296621, 332, '355'), ('916235064', 'classe_r_1027_gao__port_506270', 0.0013944063, 332, '355'), ('916235064', 'transporter_1027_gao__port_506319', 0.011966086, 332, '355'), ('916235064', 'cee_d_1027_gao__port_506288', 0.0010547738, 332, '355'), ('916235064', 'zoe_1027_gao__port_506244', 0.0020711683, 332, '355'), ('916235064', 'i20_1027_gao__port_506284', 0.0017869268, 332, '355'), ('916235064', 'gtv_1027_gao__port_506059', 0.005723421, 332, '355'), ('916235064', 's4_avant_1027_gao__port_506261', 0.002767201, 332, '355'), ('916235064', 'x1_1027_gao__port_506372', 0.0017145157, 332, '355'), ('916235064', 'autres_1027_gao__port_506127', 0.004826074, 332, '355'), ('916235064', '208_1027_gao__port_506359', 0.001868432, 332, '355'), ('916235064', 'c8_1027_gao__port_506135', 0.0012580206, 332, '355'), ('916235064', 'astra_1027_gao__port_506215', 0.001262665, 332, '355'), ('916235064', '2_1027_gao__port_506151', 0.0009245261, 332, '355'), ('916235064', 'doblo_1027_gao__port_506251', 0.007464439, 332, '355'), ('916235064', '807_1027_gao__port_506152', 0.00072908745, 332, '355'), ('916235064', '206_1027_gao__port_506126', 0.0010387457, 332, '355'), ('916235064', 'a7_1027_gao__port_506373', 0.00069124356, 332, '355'), ('916235064', 'renegade_1027_gao__port_506346', 0.0021414107, 332, '355')]]} begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 insert into MTRPhoto.class_photo_score (thcl, photo_id, hashtag_id, score) values (%s,%s,%s,%s) on duplicate key update score = values(score) time used for this insertion : 0.00013256072998046875 save missing photos in datou_result : time spend for datou_step_exec : 47.55002045631409 time spend to save output : 1.7053909301757812 total time spend for step 1 : 49.25541138648987 step2:argmax Mon Mar 17 22:37:34 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1742247405_763596_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg': 916235064} map_photo_id_path_extension : {916235064: {'path': 'temp/1742247405_763596_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou_step Argmax ! calculate argmax for thcl : 355 After datou_step_exec type output : map_portfolio_photo : len 0 keys : dict_keys([]) Inside saveOutput : final : True verbose : True photo_id : 916235064 output[photo_id] : [('916235064', 'c15_1027_gao__port_506055', 0.017712602, 332, '355'), 'temp/1742247405_763596_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg'] begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 1 insert ignore into MTRBack.photo_hashtag_ids (photo_id, hashtag_id, type) values (%s,%s,%s) insert ignore into MTRBack.photo_hashtag_ids (photo_id, hashtag_id, type) values (%s,%s,%s) first line : ('916235064', '2049863950', '332') ... last line : ('916235064', '2049863950', '332') time used for this insertion : 0.02529144287109375 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 1 insert into MTRPhoto.class_photo_score (thcl, photo_id, hashtag_id, score) values (%s,%s,%s,%s) on duplicate key update score = values(score) time used for this insertion : 0.15502405166625977 len list_finale : 1, len picture : 1 begin to insert list_values into mtr_datou_result : length of list_values in save_final : 1 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('2', None, '916235064', 'c15_1027_gao__port_506055', None, None, '2049863950', '0.017712602', None)] time used for this insertion : 0.014502525329589844 saving photo_ids in datou_result photo id not in port begin to insert list_values into mtr_datou_result : length of list_values in save_final : 0 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [] time used for this insertion : 4.291534423828125e-06 save missing photos in datou_result : time spend for datou_step_exec : 0.0032587051391601562 time spend to save output : 0.19681549072265625 total time spend for step 2 : 0.2000741958618164 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 2 output : {'916235064': [('916235064', 'c15_1027_gao__port_506055', 0.017712602, 332, '355'), 'temp/1742247405_763596_916235064_6293d1bb790dc6902450e7c572b7d10b.jpg']} ############################### TEST tfhub2 ################################ TEST TFHUB2 ######################## test with use_multi_inputs=0 ######################## Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4567 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4567 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4567 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4567 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : step 12835 tfhub_classification2 is not linked in the step_by_step architecture ! WARNING : step 12836 argmax is not linked in the step_by_step architecture ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! no param json to modify List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1171252784,1171252764,1171252487) Found this number of photos: 3 ##### Call download_photos : nb_thread : 5 begin to download photo : 1171252487 begin to download photo : 1171252764 begin to download photo : 1171252784 download finish for photo 1171252764 download finish for photo 1171252487 download finish for photo 1171252784 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 3 ; length of list_pids : 3 ; length of list_args : 3 ##### After load_data_input time to download the photos : 0.201035737991333 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 2 step1:tfhub_classification2 Mon Mar 17 22:37:35 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1742247455_763596_1171252764_29d5179a892cc50aadc9d67245534b59.jpg': 1171252764, 'temp/1742247455_763596_1171252487_5ebdd6b0a6bb39942a3808ed114806de.jpg': 1171252487, 'temp/1742247455_763596_1171252784_5a3c5d3bb155a7a116f67ded51bffb59.jpg': 1171252784} map_photo_id_path_extension : {1171252764: {'path': 'temp/1742247455_763596_1171252764_29d5179a892cc50aadc9d67245534b59.jpg', 'extension': 'jpg'}, 1171252487: {'path': 'temp/1742247455_763596_1171252487_5ebdd6b0a6bb39942a3808ed114806de.jpg', 'extension': 'jpg'}, 1171252784: {'path': 'temp/1742247455_763596_1171252784_5a3c5d3bb155a7a116f67ded51bffb59.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou_step TFHub with tf2 ! multi_thcl or not :False multi_thcl_cond or not :False dic_thcl : {'3609': 1} we are using the classfication for only one thcl 3609 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1428 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1428 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1428 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1428 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1428 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 1428 wait 20 seconds 2025-03-17 22:39:46.639908: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-17 22:39:46.641323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-17 22:39:46.641422: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-17 22:39:46.641486: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-17 22:39:46.659614: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-17 22:39:46.659839: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-17 22:39:46.692339: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-17 22:39:46.697709: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-17 22:39:46.755609: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-17 22:39:46.756708: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-17 22:39:46.758578: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-17 22:39:46.807083: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-17 22:39:46.808604: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fc3a4000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-17 22:39:46.808652: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-17 22:39:46.830993: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x82143f70 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-17 22:39:46.831047: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-17 22:39:46.835027: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-17 22:39:46.835168: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-17 22:39:46.835192: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-17 22:39:46.835303: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-17 22:39:46.835336: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-17 22:39:46.835376: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-17 22:39:46.835420: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-17 22:39:46.835465: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-17 22:39:46.836397: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-17 22:39:46.840120: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-17 22:39:46.840323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-17 22:39:46.840337: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-17 22:39:46.840345: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-17 22:39:46.841488: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3096 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) l 3637 free memory gpu now : 1428 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3609 To do loadFromThcl(), then load ParamDescType : thcl3609 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (3609) thcls : [{'id': 3609, 'mtr_user_id': 31, 'name': 'tfhub_19_06_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'jrm,pcm,pcnc,pehd,tapis_vide', 'svm_portfolios_learning': '9336903,9336904,9336905,9336906,9336909', 'photo_hashtag_type': 4674, 'photo_desc_type': 5832, 'type_classification': 'tf_classification2', 'hashtag_id_list': '495916461,560181804,1284539308,628944319,2107748999'}] thcl {'id': 3609, 'mtr_user_id': 31, 'name': 'tfhub_19_06_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'jrm,pcm,pcnc,pehd,tapis_vide', 'svm_portfolios_learning': '9336903,9336904,9336905,9336906,9336909', 'photo_hashtag_type': 4674, 'photo_desc_type': 5832, 'type_classification': 'tf_classification2', 'hashtag_id_list': '495916461,560181804,1284539308,628944319,2107748999'} Update svm_hashtag_type_desc : 5832 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (5832) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5832, 'tfhub_19_06_2023', 1280, 1280, 'tfhub_19_06_2023', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 6, 19, 12, 55, 22), datetime.datetime(2023, 6, 19, 12, 55, 22)) model_name : tfhub_19_06_2023 model_param file didn't exist model_name : tfhub_19_06_2023 model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] 2025-03-17 22:39:53.692098: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.02G (3246391296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:39:53.692693: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.72G (2921752064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:39:53.693227: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.45G (2629576704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:39:53.693730: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.20G (2366618880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:39:53.694259: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.98G (2129957120 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:39:53.694759: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.79G (1916961536 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:39:53.695314: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.61G (1725265408 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-17 22:39:53.695922: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.45G (1552738816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/tfhub_19_06_2023 /data/models_weight/tfhub_19_06_2023/Confusion_Matrix.png size_local : 57753 size in s3 : 57753 create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:15 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_jrm.jpg size_local : 79724 size in s3 : 79724 create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:20 Precision_Recall_jrm.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_pcm.jpg size_local : 83556 size in s3 : 83556 create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:15 Precision_Recall_pcm.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_pcnc.jpg size_local : 74107 size in s3 : 74107 create time local : 2023-06-22 17:09:38 create time in s3 : 2023-06-19 10:55:20 Precision_Recall_pcnc.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_pehd.jpg size_local : 72705 size in s3 : 72705 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:20 Precision_Recall_pehd.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Precision_Recall_tapis_vide.jpg size_local : 70874 size in s3 : 70874 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:15 Precision_Recall_tapis_vide.jpg already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/Result_Summary.txt size_local : 642 size in s3 : 642 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:22 Result_Summary.txt already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:22 checkpoint already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216488 size in s3 : 216488 create time local : 2023-06-22 17:09:39 create time in s3 : 2023-06-19 10:55:22 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279708 size in s3 : 32279708 create time local : 2023-06-22 17:09:40 create time in s3 : 2023-06-19 10:55:21 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/model_checkpoint.ckpt.index size_local : 43546 size in s3 : 43546 create time local : 2023-06-22 17:09:40 create time in s3 : 2023-06-19 10:55:22 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/tfhub_19_06_2023/model_weights.h5 size_local : 16499144 size in s3 : 16499144 create time local : 2023-06-22 17:09:40 create time in s3 : 2023-06-19 10:55:15 model_weights.h5 already exist and didn't need to update ERROR in datou_step_exec, will save and exit ! assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3138, in datou_step_tfhub2 this_model = model_evaluator(model_name, model_type=model_type, fc_size=fc_size,use_multi_inputs=use_multi_inputs) File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 156, in __init__ self.model, _, _ = create_tfhub_model(module_handle=self.tfhub_module, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 77, in create_tfhub_model hub.KerasLayer(module_handle, trainable=do_fine_tuning, name="module"), File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 152, in __init__ self._func = load_module(handle, tags, self._load_options) File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 421, in load_module return module_v2.load(handle, tags=tags, options=set_load_options) File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/module_v2.py", line 106, in load obj = tf.compat.v1.saved_model.load_v2(module_path, tags=tags) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 578, in load return load_internal(export_dir, tags) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 602, in load_internal loader = loader_cls(object_graph_proto, File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 123, in __init__ self._load_all() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 134, in _load_all self._load_nodes() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 264, in _load_nodes node, setter = self._recreate(proto, node_id) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 370, in _recreate return factory[kind]() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 363, in "variable": lambda: self._recreate_variable(proto.variable), File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 426, in _recreate_variable return variables.Variable( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 261, in __call__ return cls._variable_v2_call(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 243, in _variable_v2_call return previous_getter( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 66, in getter return captured_getter(captured_previous, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 418, in uninitialized_variable_creator return resource_variable_ops.UninitializedVariable(**kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 263, in __call__ return super(VariableMetaclass, cls).__call__(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 1795, in __init__ handle = _variable_handle_from_shape_and_dtype( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 174, in _variable_handle_from_shape_and_dtype gen_logging_ops._assert( # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gen_logging_ops.py", line 55, in _assert _ops.raise_from_not_ok_status(e, name) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status six.raise_from(core._status_to_exception(e.code, message), None) File "", line 3, in raise_from [1171252764, 1171252487, 1171252784] map_info['map_portfolio_photo'] : {} final : True mtd_id 4567 list_pids : [1171252764, 1171252487, 1171252784] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4567', None, '1171252764', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None), ('4567', None, '1171252487', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None), ('4567', None, '1171252784', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None)] time used for this insertion : 0.05737757682800293 save_final ERROR in last step tfhub_classification2, assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse time spend for datou_step_exec : 139.5194799900055 time spend to save output : 0.06273341178894043 total time spend for step 0 : 139.58221340179443 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 2 output : None probably due to empty image bug ERROR expected : {'1171252784': [(1171252784, 'jrm', 0.9677492, 4674, '3609'), 'temp/1687511175_1882837_1171252784_5a3c5d3bb155a7a116f67ded51bffb59.jpg'], '1171252764': [(1171252764, 'jrm', 0.9853587, 4674, '3609'), 'temp/1687511175_1882837_1171252764_29d5179a892cc50aadc9d67245534b59.jpg'], '1171252487': [(1171252487, 'jrm', 0.9262757, 4674, '3609'), 'temp/1687511175_1882837_1171252487_5ebdd6b0a6bb39942a3808ed114806de.jpg']} got : None ######################## test with use_multi_inputs=1 ######################## Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=4621 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=4621 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 4621 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=4621 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : step 12927 tfhub_classification2 is not linked in the step_by_step architecture ! WARNING : step 12928 argmax is not linked in the step_by_step architecture ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! no param json to modify List Step Type Loaded in datou : tfhub_classification2, argmax list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1171291875,1171275372,1171275314) Found this number of photos: 3 ##### Call download_photos : nb_thread : 5 begin to download photo : 1171275314 begin to download photo : 1171275372 begin to download photo : 1171291875 download finish for photo 1171275372 download finish for photo 1171291875 download finish for photo 1171275314 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 3 ; length of list_pids : 3 ; length of list_args : 3 ##### After load_data_input time to download the photos : 0.20567059516906738 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 2 step1:tfhub_classification2 Mon Mar 17 22:39:55 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1742247594_763596_1171275372_76d81364ff7df843bff095f45c07ba35.jpg': 1171275372, 'temp/1742247594_763596_1171291875_b62cd9e0d976b143f86fe82d072798c0.jpg': 1171291875, 'temp/1742247594_763596_1171275314_6e0a72c8fa00d5e4b018bd689b547133.jpg': 1171275314} map_photo_id_path_extension : {1171275372: {'path': 'temp/1742247594_763596_1171275372_76d81364ff7df843bff095f45c07ba35.jpg', 'extension': 'jpg'}, 1171291875: {'path': 'temp/1742247594_763596_1171291875_b62cd9e0d976b143f86fe82d072798c0.jpg', 'extension': 'jpg'}, 1171275314: {'path': 'temp/1742247594_763596_1171275314_6e0a72c8fa00d5e4b018bd689b547133.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou_step TFHub with tf2 ! multi_thcl or not :False multi_thcl_cond or not :False dic_thcl : {'3655': 1} we are using the classfication for only one thcl 3655 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 6 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 6 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 6 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 6 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 6 wait 20 seconds inside check gpu memory havn't enough memory gpu , need / 3096 l 3632 free memory gpu now : 6 wait 20 seconds l 3637 free memory gpu now : 6 max_wait_temp : 6 max_wait : 5 1 Physical GPUs, 1 Logical GPUs tagging for thcl : 3655 To do loadFromThcl(), then load ParamDescType : thcl3655 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (3655) thcls : [{'id': 3655, 'mtr_user_id': 31, 'name': 'tfhub_18_7_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'pcm,pcnc,jrm,pehd,tapis_vide', 'svm_portfolios_learning': '9336904,9336905,9336903,9336906,9336909', 'photo_hashtag_type': 4723, 'photo_desc_type': 5862, 'type_classification': 'tf_classification2', 'hashtag_id_list': '560181804,1284539308,495916461,628944319,2107748999'}] thcl {'id': 3655, 'mtr_user_id': 31, 'name': 'tfhub_18_7_2023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'pcm,pcnc,jrm,pehd,tapis_vide', 'svm_portfolios_learning': '9336904,9336905,9336903,9336906,9336909', 'photo_hashtag_type': 4723, 'photo_desc_type': 5862, 'type_classification': 'tf_classification2', 'hashtag_id_list': '560181804,1284539308,495916461,628944319,2107748999'} Update svm_hashtag_type_desc : 5862 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (5862) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5862, 'tfhub_18_7_2023', 1280, 1280, 'tfhub_18_7_2023', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 7, 18, 22, 46, 29), datetime.datetime(2023, 7, 18, 22, 46, 29)) model_name : tfhub_18_7_2023 model_param file didn't exist model_name : tfhub_18_7_2023 model_type : tf_classification2 list file need : ['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file exist in s3 : ['Confusion_Matrix.png', 'Precision_Recall_jrm.jpg', 'Precision_Recall_pcm.jpg', 'Precision_Recall_pcnc.jpg', 'Precision_Recall_pehd.jpg', 'Precision_Recall_tapis_vide.jpg', 'Result_Summary.txt', 'checkpoint', 'model_checkpoint.ckpt.data-00000-of-00002', 'model_checkpoint.ckpt.data-00001-of-00002', 'model_checkpoint.ckpt.index', 'model_weights.h5'] file manque in s3 : [] local folder : /data/models_weight/tfhub_18_7_2023 /data/models_weight/tfhub_18_7_2023/Confusion_Matrix.png size_local : 54360 size in s3 : 54360 create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:28 Confusion_Matrix.png already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_jrm.jpg size_local : 72583 size in s3 : 72583 create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:23 Precision_Recall_jrm.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_pcm.jpg size_local : 81681 size in s3 : 81681 create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:17 Precision_Recall_pcm.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_pcnc.jpg size_local : 79510 size in s3 : 79510 create time local : 2023-08-11 11:22:56 create time in s3 : 2023-07-18 20:46:23 Precision_Recall_pcnc.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_pehd.jpg size_local : 59936 size in s3 : 59936 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:23 Precision_Recall_pehd.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Precision_Recall_tapis_vide.jpg size_local : 78974 size in s3 : 78974 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:17 Precision_Recall_tapis_vide.jpg already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/Result_Summary.txt size_local : 642 size in s3 : 642 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:23 Result_Summary.txt already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/checkpoint size_local : 99 size in s3 : 99 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:23 checkpoint already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/model_checkpoint.ckpt.data-00000-of-00002 size_local : 216529 size in s3 : 216529 create time local : 2023-08-11 11:22:57 create time in s3 : 2023-07-18 20:46:17 model_checkpoint.ckpt.data-00000-of-00002 already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/model_checkpoint.ckpt.data-00001-of-00002 size_local : 32279748 size in s3 : 32279748 create time local : 2023-08-11 11:22:58 create time in s3 : 2023-07-18 20:46:19 model_checkpoint.ckpt.data-00001-of-00002 already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/model_checkpoint.ckpt.index size_local : 43546 size in s3 : 43546 create time local : 2023-08-11 11:22:58 create time in s3 : 2023-07-18 20:46:19 model_checkpoint.ckpt.index already exist and didn't need to update /data/models_weight/tfhub_18_7_2023/model_weights.h5 size_local : 16500868 size in s3 : 16500868 create time local : 2023-08-11 11:22:58 create time in s3 : 2023-07-18 20:46:18 model_weights.h5 already exist and didn't need to update ERROR in datou_step_exec, will save and exit ! assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2329, in datou_exec output = datou_step_exec(sNext, args, cache, context, map_info, verbose, mtr_user_id) File "/home/admin/workarea/git/Velours/python/mtr/datou/datou_lib.py", line 2523, in datou_step_exec return lib_process.datou_step_tfhub2(param, json_param, args, cache, context, map_info, verbose) File "/home/admin/workarea/git/Velours/python/mtr/datou/lib_step_exec/lib_step_process.py", line 3138, in datou_step_tfhub2 this_model = model_evaluator(model_name, model_type=model_type, fc_size=fc_size,use_multi_inputs=use_multi_inputs) File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 156, in __init__ self.model, _, _ = create_tfhub_model(module_handle=self.tfhub_module, File "/home/admin/workarea/git/Velours/python/mtr/tfhub2/evaluate.py", line 62, in create_tfhub_model fe_layer = hub.KerasLayer(module_handle, trainable=do_fine_tuning, name="module", File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 152, in __init__ self._func = load_module(handle, tags, self._load_options) File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/keras_layer.py", line 421, in load_module return module_v2.load(handle, tags=tags, options=set_load_options) File "/home/admin/.local/lib/python3.8/site-packages/tensorflow_hub/module_v2.py", line 106, in load obj = tf.compat.v1.saved_model.load_v2(module_path, tags=tags) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 578, in load return load_internal(export_dir, tags) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 602, in load_internal loader = loader_cls(object_graph_proto, File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 123, in __init__ self._load_all() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 134, in _load_all self._load_nodes() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 264, in _load_nodes node, setter = self._recreate(proto, node_id) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 370, in _recreate return factory[kind]() File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 363, in "variable": lambda: self._recreate_variable(proto.variable), File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 426, in _recreate_variable return variables.Variable( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 261, in __call__ return cls._variable_v2_call(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 243, in _variable_v2_call return previous_getter( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 66, in getter return captured_getter(captured_previous, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 418, in uninitialized_variable_creator return resource_variable_ops.UninitializedVariable(**kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/variables.py", line 263, in __call__ return super(VariableMetaclass, cls).__call__(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 1795, in __init__ handle = _variable_handle_from_shape_and_dtype( File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 174, in _variable_handle_from_shape_and_dtype gen_logging_ops._assert( # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gen_logging_ops.py", line 55, in _assert _ops.raise_from_not_ok_status(e, name) File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status six.raise_from(core._status_to_exception(e.code, message), None) File "", line 3, in raise_from [1171275372, 1171291875, 1171275314] map_info['map_portfolio_photo'] : {} final : True mtd_id 4621 list_pids : [1171275372, 1171291875, 1171275314] begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('4621', None, '1171275372', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None), ('4621', None, '1171291875', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None), ('4621', None, '1171275314', "[>, , , , , 'assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse']", '-1', '-1.0', '501120777', '1.0', None)] time used for this insertion : 0.014734745025634766 save_final ERROR in last step tfhub_classification2, assertion failed: [0] [Op:Assert] name: EagerVariableNameReuse time spend for datou_step_exec : 133.90437412261963 time spend to save output : 0.015355348587036133 total time spend for step 0 : 133.91972947120667 need to delete datou_research and reload, so keep current state 1 caffe_path_current : About to save ! 2 After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 2 output : None probably due to empty image bug ERROR expected : {'1171291875': [(1171291875, 'tapis_vide', 0.97062814, 4723, '3655'), 'temp/1691745841_1143057_1171291875_b62cd9e0d976b143f86fe82d072798c0.jpg'], '1171275372': [(1171275372, 'tapis_vide', 0.9674145, 4723, '3655'), 'temp/1691745841_1143057_1171275372_76d81364ff7df843bff095f45c07ba35.jpg'], '1171275314': [(1171275314, 'tapis_vide', 0.96509415, 4723, '3655'), 'temp/1691745841_1143057_1171275314_6e0a72c8fa00d5e4b018bd689b547133.jpg']} got : None ERROR tfhub2 FAILED ############################### TEST ordonner ################################ To do loadFromThcl(), then load ParamDescType : thcl358 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (358) thcls : [{'id': 358, 'mtr_user_id': 31, 'name': 'car_orientation_0111', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'FirstUploadExperveo_vignette__port_505674,CAR_EXTERIEUR_Roue__port_503398,FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486,FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485,CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465,CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198,CAR_EXTERIEUR_Face_avant_axe_droit__port_504451,CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235,FirstUploadExperveo_vin__port_505675,CAR_EXTERIEUR_cote_droite__port_504108,CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565,FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483,CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201,cartegrise_orientation__port_505064,CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217,CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531,CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218,CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214,CAR_EXTERIEUR_Angle_avant_droit__port_504087,FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484,CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563,CAR_EXTERIEUR_Angle_arriere_droit__port_504160,CAR_EXTERIEUR_arriere__port_504184,CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562,INTERIEUR_Compteur_kilometrique__port_503644,CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494,CAR_EXTERIEUR_Angle_arriere_gauche__port_504170,CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226,CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202,CAR_EXTERIEUR_moteur__port_503704,FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487,CAR_INTERIEUR_siege_arriere_class_1__port_506551,CAR_EXTERIEUR_avant__port_504146,CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215,CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225,CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564,FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482,CAR_INTERIEUR_coffre__port_503412,FirstUploadExperveo_rouetranche__port_505677,UploadPhotoImmatBest_class_1__port_505051,CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532,CAR_EXTERIEUR_angle_avant_gauche__port_504098,CAR_EXTERIEUR_face_avant_axe_gauche__port_504236,CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540,CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233,CAR_EXTERIEUR_roue_de_secour__port_503763,CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199,CAR_EXTERIEUR_cote_gauche__port_504017,CAR_INTERIEUR_avant_volant_class_1__port_506503,CAR_INTERIEUR_avant_volant_class_2__port_506504,CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234', 'svm_portfolios_learning': '505674,503398,506486,506485,504465,504198,504451,504235,505675,504108,506565,506483,504201,505064,504217,506531,504218,504214,504087,506484,506563,504160,504184,506562,503644,506494,504170,504226,504202,503704,506487,506551,504146,504215,504225,506564,506482,503412,505677,505051,506532,504098,504236,506540,504233,503763,504199,504017,506503,506504,504234', 'photo_hashtag_type': 337, 'photo_desc_type': 3392, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 358, 'mtr_user_id': 31, 'name': 'car_orientation_0111', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'FirstUploadExperveo_vignette__port_505674,CAR_EXTERIEUR_Roue__port_503398,FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486,FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485,CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465,CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198,CAR_EXTERIEUR_Face_avant_axe_droit__port_504451,CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235,FirstUploadExperveo_vin__port_505675,CAR_EXTERIEUR_cote_droite__port_504108,CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565,FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483,CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201,cartegrise_orientation__port_505064,CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217,CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531,CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218,CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214,CAR_EXTERIEUR_Angle_avant_droit__port_504087,FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484,CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563,CAR_EXTERIEUR_Angle_arriere_droit__port_504160,CAR_EXTERIEUR_arriere__port_504184,CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562,INTERIEUR_Compteur_kilometrique__port_503644,CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494,CAR_EXTERIEUR_Angle_arriere_gauche__port_504170,CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226,CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202,CAR_EXTERIEUR_moteur__port_503704,FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487,CAR_INTERIEUR_siege_arriere_class_1__port_506551,CAR_EXTERIEUR_avant__port_504146,CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215,CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225,CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564,FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482,CAR_INTERIEUR_coffre__port_503412,FirstUploadExperveo_rouetranche__port_505677,UploadPhotoImmatBest_class_1__port_505051,CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532,CAR_EXTERIEUR_angle_avant_gauche__port_504098,CAR_EXTERIEUR_face_avant_axe_gauche__port_504236,CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540,CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233,CAR_EXTERIEUR_roue_de_secour__port_503763,CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199,CAR_EXTERIEUR_cote_gauche__port_504017,CAR_INTERIEUR_avant_volant_class_1__port_506503,CAR_INTERIEUR_avant_volant_class_2__port_506504,CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234', 'svm_portfolios_learning': '505674,503398,506486,506485,504465,504198,504451,504235,505675,504108,506565,506483,504201,505064,504217,506531,504218,504214,504087,506484,506563,504160,504184,506562,503644,506494,504170,504226,504202,503704,506487,506551,504146,504215,504225,506564,506482,503412,505677,505051,506532,504098,504236,506540,504233,503763,504199,504017,506503,506504,504234', 'photo_hashtag_type': 337, 'photo_desc_type': 3392, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 3392 ['FirstUploadExperveo_vignette__port_505674', 'CAR_EXTERIEUR_Roue__port_503398', 'FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486', 'FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485', 'CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465', 'CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198', 'CAR_EXTERIEUR_Face_avant_axe_droit__port_504451', 'CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235', 'FirstUploadExperveo_vin__port_505675', 'CAR_EXTERIEUR_cote_droite__port_504108', 'CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565', 'FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483', 'CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201', 'cartegrise_orientation__port_505064', 'CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217', 'CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531', 'CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218', 'CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214', 'CAR_EXTERIEUR_Angle_avant_droit__port_504087', 'FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484', 'CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563', 'CAR_EXTERIEUR_Angle_arriere_droit__port_504160', 'CAR_EXTERIEUR_arriere__port_504184', 'CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562', 'INTERIEUR_Compteur_kilometrique__port_503644', 'CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494', 'CAR_EXTERIEUR_Angle_arriere_gauche__port_504170', 'CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226', 'CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202', 'CAR_EXTERIEUR_moteur__port_503704', 'FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487', 'CAR_INTERIEUR_siege_arriere_class_1__port_506551', 'CAR_EXTERIEUR_avant__port_504146', 'CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215', 'CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225', 'CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564', 'FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482', 'CAR_INTERIEUR_coffre__port_503412', 'FirstUploadExperveo_rouetranche__port_505677', 'UploadPhotoImmatBest_class_1__port_505051', 'CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532', 'CAR_EXTERIEUR_angle_avant_gauche__port_504098', 'CAR_EXTERIEUR_face_avant_axe_gauche__port_504236', 'CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540', 'CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233', 'CAR_EXTERIEUR_roue_de_secour__port_503763', 'CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199', 'CAR_EXTERIEUR_cote_gauche__port_504017', 'CAR_INTERIEUR_avant_volant_class_1__port_506503', 'CAR_INTERIEUR_avant_volant_class_2__port_506504', 'CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234'] 51 SELECT hashtag_id,hashtag FROM MTRBack.hashtags where hashtag in ('FirstUploadExperveo_vignette__port_505674','CAR_EXTERIEUR_Roue__port_503398','FirstUploadExperveo_carrosseriegrosplan_VIndanslamoquette__port_506486','FirstUploadExperveo_carrosseriegrosplan_siegegrosplan__port_506485','CAR_EXTERIEUR_Cote_droit_axe_avant__port_504465','CAR_EXTERIEUR_Cote_gauche_axe_arriere__port_504198','CAR_EXTERIEUR_Face_avant_axe_droit__port_504451','CAR_EXTERIEUR_angle_avant_gauche_axe_avant__port_504235','FirstUploadExperveo_vin__port_505675','CAR_EXTERIEUR_cote_droite__port_504108','CAR_INTERIEUR_avant_volant_class_6_levierdevitesse__port_506565','FirstUploadExperveo_carrosseriegrosplan_carrosserie__port_506483','CAR_EXTERIEUR_Angle_arriere_gauche_axe_arriere__port_504201','cartegrise_orientation__port_505064','CAR_EXTERIEUR_Angle_arriere_droit_axe_arriere__port_504217','CAR_INTERIEUR_avant_vue-arriere_class_1__port_506531','CAR_EXTERIEUR_Face_arriere_axe_droit__port_504218','CAR_EXTERIEUR_Cote_droit_axe_arriere__port_504214','CAR_EXTERIEUR_Angle_avant_droit__port_504087','FirstUploadExperveo_carrosseriegrosplan_morceauderoue__port_506484','CAR_INTERIEUR_avant_volant_class_6_class_2__port_506563','CAR_EXTERIEUR_Angle_arriere_droit__port_504160','CAR_EXTERIEUR_arriere__port_504184','CAR_INTERIEUR_avant_volant_class_6_boutonrond__port_506562','INTERIEUR_Compteur_kilometrique__port_503644','CAR_INTERIEUR_avant_vue_gauche_habitacle_class_1__port_506494','CAR_EXTERIEUR_Angle_arriere_gauche__port_504170','CAR_EXTERIEUR_Angle_avant_droit_axe_arriere__port_504226','CAR_EXTERIEUR_Face_arriere_axe_gauche__port_504202','CAR_EXTERIEUR_moteur__port_503704','FirstUploadExperveo_carrosseriegrosplan_class_6__port_506487','CAR_INTERIEUR_siege_arriere_class_1__port_506551','CAR_EXTERIEUR_avant__port_504146','CAR_EXTERIEUR_Angle_arriere_droit_axe_droit__port_504215','CAR_EXTERIEUR_Angle_avant_droit_axe_droit__port_504225','CAR_INTERIEUR_avant_volant_class_6_ecrangrosplan__port_506564','FirstUploadExperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482','CAR_INTERIEUR_coffre__port_503412','FirstUploadExperveo_rouetranche__port_505677','UploadPhotoImmatBest_class_1__port_505051','CAR_INTERIEUR_avant_vue-arriere_class_2__port_506532','CAR_EXTERIEUR_angle_avant_gauche__port_504098','CAR_EXTERIEUR_face_avant_axe_gauche__port_504236','CAR_INTERIEUR_avant_vue_droite_habitacle_class_1__port_506540','CAR_EXTERIEUR_cote_gauche_axe_avant__port_504233','CAR_EXTERIEUR_roue_de_secour__port_503763','CAR_EXTERIEUR_Angle_arriere_gauche_axe_gauche__port_504199','CAR_EXTERIEUR_cote_gauche__port_504017','CAR_INTERIEUR_avant_volant_class_1__port_506503','CAR_INTERIEUR_avant_volant_class_2__port_506504','CAR_EXTERIEUR_angle_avant_gauche_axe_gauche__port_504234'); 51 dict_keys(['cartegrise_orientation__port_505064', 'car_exterieur_angle_arriere_droit_axe_arriere__port_504217', 'car_exterieur_angle_arriere_droit_axe_droit__port_504215', 'car_exterieur_angle_arriere_droit__port_504160', 'car_exterieur_angle_arriere_gauche_axe_arriere__port_504201', 'car_exterieur_angle_arriere_gauche_axe_gauche__port_504199', 'car_exterieur_angle_arriere_gauche__port_504170', 'car_exterieur_angle_avant_droit_axe_arriere__port_504226', 'car_exterieur_angle_avant_droit_axe_droit__port_504225', 'car_exterieur_angle_avant_droit__port_504087', 'car_exterieur_angle_avant_gauche_axe_avant__port_504235', 'car_exterieur_angle_avant_gauche_axe_gauche__port_504234', 'car_exterieur_angle_avant_gauche__port_504098', 'car_exterieur_arriere__port_504184', 'car_exterieur_avant__port_504146', 'car_exterieur_cote_droite__port_504108', 'car_exterieur_cote_droit_axe_arriere__port_504214', 'car_exterieur_cote_droit_axe_avant__port_504465', 'car_exterieur_cote_gauche_axe_arriere__port_504198', 'car_exterieur_cote_gauche_axe_avant__port_504233', 'car_exterieur_cote_gauche__port_504017', 'car_exterieur_face_arriere_axe_droit__port_504218', 'car_exterieur_face_arriere_axe_gauche__port_504202', 'car_exterieur_face_avant_axe_droit__port_504451', 'car_exterieur_face_avant_axe_gauche__port_504236', 'car_exterieur_moteur__port_503704', 'car_exterieur_roue_de_secour__port_503763', 'car_exterieur_roue__port_503398', 'car_interieur_avant_volant_class_1__port_506503', 'car_interieur_avant_volant_class_2__port_506504', 'car_interieur_avant_volant_class_6_boutonrond__port_506562', 'car_interieur_avant_volant_class_6_class_2__port_506563', 'car_interieur_avant_volant_class_6_ecrangrosplan__port_506564', 'car_interieur_avant_volant_class_6_levierdevitesse__port_506565', 'car_interieur_avant_vue-arriere_class_1__port_506531', 'car_interieur_avant_vue-arriere_class_2__port_506532', 'car_interieur_avant_vue_droite_habitacle_class_1__port_506540', 'car_interieur_avant_vue_gauche_habitacle_class_1__port_506494', 'car_interieur_coffre__port_503412', 'car_interieur_siege_arriere_class_1__port_506551', 'firstuploadexperveo_carrosseriegrosplan_carrosserie__port_506483', 'firstuploadexperveo_carrosseriegrosplan_class_6__port_506487', 'firstuploadexperveo_carrosseriegrosplan_morceauderoue__port_506484', 'firstuploadexperveo_carrosseriegrosplan_moteurgrosplanetdegat__port_506482', 'firstuploadexperveo_carrosseriegrosplan_siegegrosplan__port_506485', 'firstuploadexperveo_carrosseriegrosplan_vindanslamoquette__port_506486', 'firstuploadexperveo_rouetranche__port_505677', 'firstuploadexperveo_vignette__port_505674', 'firstuploadexperveo_vin__port_505675', 'interieur_compteur_kilometrique__port_503644', 'uploadphotoimmatbest_class_1__port_505051']) select photo_hashtag_type from MTRDatou.classification_theme where id = 358 thcl : 358 photo_hashtag_type : 337 SELECT phi.hashtag_id , phi.photo_id FROM MTRBack.photo_hashtag_ids phi, MTRUser.mtr_portfolio_photos mp where phi.type = 337 and phi.photo_id = mp.mtr_photo_id and mp.mtr_portfolio_id =510365; {510365: [(917973295, 1), (917973297, 1), (917973302, 1), (917973293, 7), (917973296, 11), (917973300, 11), (917973286, 13), (917973289, 13), (917973301, 24), (917973285, 29), (917973290, 29), (917973299, 29), (917973304, 35), (917973287, 36), (917973298, 36), (917973305, 36), (917973292, 37), (917973291, 41), (917973303, 41), (917973294, 42), (917973288, 46)]} ############################### TEST rotate ################################ test rotate only Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=230 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=230 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 230 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=230 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! We are managing only one step so we do not consider checkConsistencyNbInputNbOutput ! We are managing only one step so we do not consider checkConsistencyTypeOutputInput ! no param json to modify List Step Type Loaded in datou : rotate list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (917849322) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 917849322 download finish for photo 917849322 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.1696615219116211 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 1 step1:rotate Mon Mar 17 22:42:24 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1742247744_763596_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg': 917849322} map_photo_id_path_extension : {917849322: {'path': 'temp/1742247744_763596_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou_step_rotate ! We are in a linear step without datou_depend ! rotate photos of 90,180,270 degres batch 1 select photo_id, hashtag_id, `type`, x0, x1, y0, y1, score, id from MTRPhoto.crop_hashtag_ids where photo_id in ( 917849322) and `type` in (0) Loaded 0 chid ids of type : 0 SELECT crop_hashtag_id, points FROM MTRPhoto.crop_polygon_points WHERE crop_hashtag_id in () map_chi : {} photo_id in download_rotate_and_save : 917849322 list_chi_loc : 0 Use all angle ! Rotation of photo 917849322 of 90 degree temp/1742247744_763596_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg [] 90 remove_crop_border : False version de PIL : 9.5.0 Needs to change image size ! [[ 6.123234e-17 1.000000e+00] [-1.000000e+00 6.123234e-17]] 90 [[ 6.123234e-17 1.000000e+00] [-1.000000e+00 6.123234e-17]] shrink_image : False image_rotate : image_path : temp/1742247744_763596_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg path_name_rotate : temp/1742247744_763596_917849322_2bd260e91e91df8378dde8bb8b8c454890.jpg image_rotate.mode : RGB Rotation of photo 917849322 of 180 degree temp/1742247744_763596_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg [] 180 remove_crop_border : False version de PIL : 9.5.0 Needs to change image size ! [[-1.0000000e+00 1.2246468e-16] [-1.2246468e-16 -1.0000000e+00]] 180 [[-1.0000000e+00 1.2246468e-16] [-1.2246468e-16 -1.0000000e+00]] shrink_image : False image_rotate : image_path : temp/1742247744_763596_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg path_name_rotate : temp/1742247744_763596_917849322_2bd260e91e91df8378dde8bb8b8c4548180.jpg image_rotate.mode : RGB Rotation of photo 917849322 of 270 degree temp/1742247744_763596_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg [] 270 remove_crop_border : False version de PIL : 9.5.0 Needs to change image size ! [[-1.8369702e-16 -1.0000000e+00] [ 1.0000000e+00 -1.8369702e-16]] 270 [[-1.8369702e-16 -1.0000000e+00] [ 1.0000000e+00 -1.8369702e-16]] shrink_image : False image_rotate : image_path : temp/1742247744_763596_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg path_name_rotate : temp/1742247744_763596_917849322_2bd260e91e91df8378dde8bb8b8c4548270.jpg image_rotate.mode : RGB About to upload 3 photos upload in portfolio : 551782 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1742247745_763596 we have uploaded 3 photos in the portfolio 551782 time of upload the photos Elapsed time : 1.0178508758544922 map_filename_photo_id : 3 map_filename_photo_id : {'temp/1742247744_763596_917849322_2bd260e91e91df8378dde8bb8b8c454890.jpg': 1345014550, 'temp/1742247744_763596_917849322_2bd260e91e91df8378dde8bb8b8c4548180.jpg': 1345014551, 'temp/1742247744_763596_917849322_2bd260e91e91df8378dde8bb8b8c4548270.jpg': 1345014553} Len new_chis : 3 Len list_new_chi_with_photo_id : 0 of type : 0 list_new_chi_with_photo_id : [] After datou_step_exec type output : time spend for datou_step_exec : 1.2608013153076172 time spend to save output : 6.604194641113281e-05 total time spend for step 1 : 1.2608673572540283 caffe_path_current : About to save ! 1 Inside saveOutput : final : True verbose : True saveOutput not yet implemented for datou_step.type : rotate we use saveGeneral [917849322] map_info['map_portfolio_photo'] : {} final : True mtd_id 230 list_pids : [917849322] Looping around the photos to save general results len do output : 3 /1345014550Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1345014551Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1345014553Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('230', None, None, None, None, None, None, None, None) ('230', None, '917849322', None, None, None, None, None, None) begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 insert ignore into MTRPhoto.mtr_datou_result (mtd_id, mtr_portfolio_id,mtr_photo_id,result,result_long,result_double,hashtag_id,proba, mtr_current_id) values (%s,%s,%s,%s,%s,%s,%s,%s,%s) on duplicate key update mtr_portfolio_id = mtr_portfolio_id list_values : [('230', None, '1345014550', 'None', None, None, None, None, None), ('230', None, '1345014551', 'None', None, None, None, None, None), ('230', None, '1345014553', 'None', None, None, None, None, None), ('230', None, '917849322', None, None, None, None, None, None)] time used for this insertion : 0.014525175094604492 save_final save missing photos in datou_result : After save, about to update current ! datou_cur_ids : [] len(datou.list_steps) : 1 output : {1345014550: ['917849322', 'temp/1742247744_763596_917849322_2bd260e91e91df8378dde8bb8b8c454890.jpg', []], 1345014551: ['917849322', 'temp/1742247744_763596_917849322_2bd260e91e91df8378dde8bb8b8c4548180.jpg', []], 1345014553: ['917849322', 'temp/1742247744_763596_917849322_2bd260e91e91df8378dde8bb8b8c4548270.jpg', []]} test rotate only is a success ! test rotate conditionnel Inside batchDatouExec : verbose : True ##### chargement datou SELECT name, created_at,limit_max FROM MTRDatou.mtr_datou WHERE id=233 SELECT mtd.id, mtdt.`type`, mtd.`param`, mtd.param_json, mtdt.nb_input, mtdt.nb_output, mtdt.prod, mtdt.is_local, mtdt.is_datou_depend, mtdt.is_photo_id_local FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_step_types mtdt WHERE mtdt.`id`=mtd.`type` AND mtd.mtd_id=233 SELECT mtd.id, mtd.mtd_id, mdsdt.id, mdsdt.name, mdsdt.description, msid.output_or_input, msid.data_order_id, mdsdt.type FROM MTRDatou.mtr_datou_step mtd, MTRDatou.mtr_datou_steptype_io_datatypes msid, MTRDatou.mtr_datou_step_data_types mdsdt WHERE mtd.`type`=msid.`mtr_datou_step_type` AND mtd.mtd_id= 233 AND msid.data_type=mdsdt.id SELECT mts_id_output, id_output, mts_id_input, id_input FROM MTRDatou.mtr_datou_step_by_step WHERE mtd_id=233 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! no param json to modify List Step Type Loaded in datou : thcl, argmax, rotate list_input_json : [] ##### fin chargement datou ##### chargement data ##### Call load_data_input : nb_thread : 5 origin SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (917849322) Found this number of photos: 1 ##### Call download_photos : nb_thread : 5 begin to download photo : 917849322 download finish for photo 917849322 we have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB ##### After download_photos length of list_filenames : 1 ; length of list_pids : 1 ; length of list_args : 1 ##### After load_data_input time to download the photos : 0.10016655921936035 #### fin chargement data Blocking on flush ? No conitnuing About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : True number of steps : 3 step1:thcl Mon Mar 17 22:42:26 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec After prepare type args : Here we display some param of map_info ! map_filenames : {'temp/1742247746_763596_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg': 917849322} map_photo_id_path_extension : {917849322: {'path': 'temp/1742247746_763596_917849322_2bd260e91e91df8378dde8bb8b8c4548.jpg', 'extension': 'jpg'}} map_subphoto_mainphoto : {} Beginning of datou step Thcl ! multi_thcl or not :False multi_thcl_cond or not :False dic_thcl : {'500': 1} we are using the classfication for only one thcl 500 In convert_file_to_np l 337 : 1 l343 1 l357 after caffe.io.load_image dimension du image : (3, (2448, 3264, 3)) dimension displayed ! time to import caffe and check if the image exist : 0.00036334991455078125 time to convert the images to numpy array : 2.1753504276275635 total time to convert the images to numpy array : 2.1761116981506348 list photo_ids error: [] list photo_ids correct : [917849322] number of photos to traite : 1 try to delete the photos incorrect in DB tagging for thcl : 500 To do loadFromThcl(), then load ParamDescType : thcl500 get_desc_type_from_thcl : type of cat SELECT id, mtr_user_id, name, pb_hashtag_id, hashtag_id_list, button_legend_list, portfolio_id_lists, photo_hashtag_type, photo_desc_type, svm_limit, limit_tagging, is_public, live, created_at, updated_at, type_classification FROM MTRDatou.classification_theme WHERE `id` IN (500) thcls : [{'id': 500, 'mtr_user_id': 31, 'name': 'orientation_carte_grise_all_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'carteGrisesVerticales__port_549774,cartegrise_90deg__port_550987,cartesGrisesEnvers__port_549765,portfolio_270deg__port_550988', 'svm_portfolios_learning': '549774,550987,549765,550988', 'photo_hashtag_type': 507, 'photo_desc_type': 3517, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0'}] thcl {'id': 500, 'mtr_user_id': 31, 'name': 'orientation_carte_grise_all_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'carteGrisesVerticales__port_549774,cartegrise_90deg__port_550987,cartesGrisesEnvers__port_549765,portfolio_270deg__port_550988', 'svm_portfolios_learning': '549774,550987,549765,550988', 'photo_hashtag_type': 507, 'photo_desc_type': 3517, 'type_classification': 'caffe', 'hashtag_id_list': '0,0,0,0'} Update svm_hashtag_type_desc : 3517 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (3517) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3517, 'orientation_carte_grise_all_2', 16384, 25088, 'orientation_carte_grise_all_2', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 4, 18, 20, 4, 34), datetime.datetime(2018, 4, 18, 20, 4, 34)) To loadFromThcl() : net_3517 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 6 wait 20 seconds l 3637 free memory gpu now : 6 max_wait_temp : 1 max_wait : 0 SELECT * FROM MTRDatou.photo_desc_type_params WHERE id in (3517) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (3517, 'orientation_carte_grise_all_2', 16384, 25088, 'orientation_carte_grise_all_2', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2018, 4, 18, 20, 4, 34), datetime.datetime(2018, 4, 18, 20, 4, 34)) param : , param.caffemodel : orientation_carte_grise_all_2 None mean_file_type : mean_file_path : prototxt_file_path : model : orientation_carte_grise_all_2 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : orientation_carte_grise_all_2 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : [] local folder : /data/models_weight/orientation_carte_grise_all_2 /data/models_weight/orientation_carte_grise_all_2/caffemodel size_local : 537110520 size in s3 : 537110520 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:17 caffemodel already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/deploy_conv_normal.prototxt size_local : 4626 size in s3 : 4626 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16 deploy_conv_normal.prototxt already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/deploy_fc.prototxt size_local : 1130 size in s3 : 1130 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16 deploy_fc.prototxt already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/deploy.prototxt size_local : 5653 size in s3 : 5653 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16 deploy.prototxt already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:31 mean.npy already exist and didn't need to update /data/models_weight/orientation_carte_grise_all_2/synset_words.txt size_local : 159 size in s3 : 159 create time local : 2021-08-09 05:29:00 create time in s3 : 2021-08-06 20:07:16 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/orientation_carte_grise_all_2/deploy.prototxt caffemodel_filename : /data/models_weight/orientation_carte_grise_all_2/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 6 wait 20 seconds WARNING: Logging before InitGoogleLogging() is written to STDERR F0317 22:43:18.293411 763596 syncedmem.cpp:78] Check failed: error == cudaSuccess (2 vs. 0) out of memory *** Check failure stack trace: *** Command terminated by signal 6 43.60user 36.36system 1:07:53elapsed 1%CPU (0avgtext+0avgdata 3977584maxresident)k 10071864inputs+22744outputs (56097major+3549084minor)pagefaults 0swaps