python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 4234 -l 5 ' -s datou_current_4234 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 494175 load datou : 4234 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5275, 'learn_RUBBIA_REFUS_AMIENS_23', 16384, 25088, 'learn_RUBBIA_REFUS_AMIENS_23', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 23, 14, 19, 39), datetime.datetime(2021, 4, 23, 14, 19, 39)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5561, 'learn_rubbia_refus_2500', 2048, 2048, 'learn_rubbia_refus_2500', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2021, 12, 2, 19, 10, 8), datetime.datetime(2021, 12, 2, 19, 10, 8)) load thcls load THCL from format json or kwargs add thcl : 2847 in CacheModelConfig load THCL from format json or kwargs add thcl : 3237 in CacheModelConfig load pdts add pdt : 5275 in CacheModelConfig add pdt : 5561 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 list_input_json: [] Current got : datou_id : 4234, datou_cur_ids : ['2612493'] with mtr_portfolio_ids : ['20800049'] and first list_photo_ids : [] new path : /proc/494175/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, brightness, blur_detection, rle_unique_nms_with_priority, crop_condition, thcl, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , BFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 2 ; length of list_pids : 2 ; length of list_args : 2 time to download the photos : 0.2230675220489502 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Tue Feb 25 11:20:28 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10593 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-25 11:20:31.764307: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-25 11:20:31.791314: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-25 11:20:31.793573: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7efd70000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-25 11:20:31.793628: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-25 11:20:31.797584: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-25 11:20:31.937513: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x367f0450 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-25 11:20:31.937559: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-25 11:20:31.938828: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-25 11:20:31.939351: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-25 11:20:31.944967: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-25 11:20:31.947350: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-25 11:20:31.947930: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-25 11:20:31.950588: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-25 11:20:31.953656: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-25 11:20:31.973911: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-25 11:20:31.976737: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-25 11:20:31.976884: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-25 11:20:31.980261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-25 11:20:31.980309: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-25 11:20:31.980323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-25 11:20:31.995304: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9671 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-25 11:20:32.292118: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-25 11:20:32.292273: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-25 11:20:32.292321: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-25 11:20:32.292368: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-25 11:20:32.292406: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-25 11:20:32.292433: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-25 11:20:32.292457: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-25 11:20:32.292481: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-25 11:20:32.294259: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-25 11:20:32.296653: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-25 11:20:32.296778: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-25 11:20:32.296796: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-25 11:20:32.296812: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-25 11:20:32.296827: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-25 11:20:32.296842: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-25 11:20:32.296856: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-25 11:20:32.296871: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-25 11:20:32.298330: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-25 11:20:32.298377: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-25 11:20:32.298388: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-25 11:20:32.298396: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-25 11:20:32.299828: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9671 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2847 thcls : [{'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}] thcl {'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5275 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5275, 'learn_RUBBIA_REFUS_AMIENS_23', 16384, 25088, 'learn_RUBBIA_REFUS_AMIENS_23', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 23, 14, 19, 39), datetime.datetime(2021, 4, 23, 14, 19, 39)) {'thcl': {'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'], 'list_hashtags_csv': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'svm_hashtag_type_desc': 5275, 'photo_desc_type': 5275, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_RUBBIA_REFUS_AMIENS_23 NUM_CLASSES 9 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_RUBBIA_REFUS_AMIENS_23 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-25 11:20:42.195950: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-25 11:20:42.402079: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-25 11:20:44.080899: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/learn_RUBBIA_REFUS_AMIENS_23 /data/models_weight/learn_RUBBIA_REFUS_AMIENS_23/mask_model.h5 size_local : 256009536 size in s3 : 256009536 create time local : 2021-08-09 09:43:22 create time in s3 : 2021-08-06 18:54:04 mask_model.h5 already exist and didn't need to update list_images length : 2 NEW PHOTO pour l'instant on ne peut pas sauvegarder la photo dans les tile Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -119.55547 max: 145.71328 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 2 time for calcul the mask position with numpy : 0.00011539459228515625 nb_pixel_total : 1066 time to create 1 rle with old method : 0.0020017623901367188 length of segment : 35 time for calcul the mask position with numpy : 8.344650268554688e-05 nb_pixel_total : 1241 time to create 1 rle with old method : 0.006878376007080078 length of segment : 32 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -115.63750 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 11 time for calcul the mask position with numpy : 9.655952453613281e-05 nb_pixel_total : 698 time to create 1 rle with old method : 0.0014793872833251953 length of segment : 33 time for calcul the mask position with numpy : 0.00013709068298339844 nb_pixel_total : 2377 time to create 1 rle with old method : 0.0050334930419921875 length of segment : 47 time for calcul the mask position with numpy : 6.961822509765625e-05 nb_pixel_total : 253 time to create 1 rle with old method : 0.0004825592041015625 length of segment : 30 time for calcul the mask position with numpy : 0.0001049041748046875 nb_pixel_total : 1079 time to create 1 rle with old method : 0.0018613338470458984 length of segment : 36 time for calcul the mask position with numpy : 0.000125885009765625 nb_pixel_total : 2332 time to create 1 rle with old method : 0.0038461685180664062 length of segment : 98 time for calcul the mask position with numpy : 7.43865966796875e-05 nb_pixel_total : 202 time to create 1 rle with old method : 0.0004100799560546875 length of segment : 19 time for calcul the mask position with numpy : 0.00010800361633300781 nb_pixel_total : 1977 time to create 1 rle with old method : 0.0035600662231445312 length of segment : 61 time for calcul the mask position with numpy : 9.274482727050781e-05 nb_pixel_total : 198 time to create 1 rle with old method : 0.00038814544677734375 length of segment : 18 time for calcul the mask position with numpy : 0.0001373291015625 nb_pixel_total : 2297 time to create 1 rle with old method : 0.004404783248901367 length of segment : 47 time for calcul the mask position with numpy : 0.00013685226440429688 nb_pixel_total : 1756 time to create 1 rle with old method : 0.0035965442657470703 length of segment : 65 time for calcul the mask position with numpy : 0.0001723766326904297 nb_pixel_total : 1533 time to create 1 rle with old method : 0.0033528804779052734 length of segment : 59 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -103.90000 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 5 time for calcul the mask position with numpy : 0.00017142295837402344 nb_pixel_total : 5722 time to create 1 rle with old method : 0.009800195693969727 length of segment : 100 time for calcul the mask position with numpy : 6.556510925292969e-05 nb_pixel_total : 751 time to create 1 rle with old method : 0.0015091896057128906 length of segment : 44 time for calcul the mask position with numpy : 0.00010061264038085938 nb_pixel_total : 3202 time to create 1 rle with old method : 0.0059545040130615234 length of segment : 45 time for calcul the mask position with numpy : 5.888938903808594e-05 nb_pixel_total : 540 time to create 1 rle with old method : 0.0010905265808105469 length of segment : 26 time for calcul the mask position with numpy : 0.00012731552124023438 nb_pixel_total : 1482 time to create 1 rle with old method : 0.003246307373046875 length of segment : 62 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -103.90000 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 2 time for calcul the mask position with numpy : 0.00016307830810546875 nb_pixel_total : 5964 time to create 1 rle with old method : 0.010809183120727539 length of segment : 93 time for calcul the mask position with numpy : 0.0001423358917236328 nb_pixel_total : 932 time to create 1 rle with old method : 0.002429485321044922 length of segment : 46 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 4 time for calcul the mask position with numpy : 0.0001163482666015625 nb_pixel_total : 1435 time to create 1 rle with old method : 0.0036306381225585938 length of segment : 108 time for calcul the mask position with numpy : 0.00014400482177734375 nb_pixel_total : 2419 time to create 1 rle with old method : 0.005209922790527344 length of segment : 62 time for calcul the mask position with numpy : 7.700920104980469e-05 nb_pixel_total : 517 time to create 1 rle with old method : 0.0012755393981933594 length of segment : 19 time for calcul the mask position with numpy : 7.915496826171875e-05 nb_pixel_total : 677 time to create 1 rle with old method : 0.00162506103515625 length of segment : 62 Processing 1 images image shape: (400, 400, 3) min: 21.00000 max: 201.00000 molded_images shape: (1, 640, 640, 3) min: -88.66094 max: 73.60391 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 0 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 252.00000 molded_images shape: (1, 640, 640, 3) min: -104.75859 max: 127.78750 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 3 time for calcul the mask position with numpy : 8.654594421386719e-05 nb_pixel_total : 325 time to create 1 rle with old method : 0.0007445812225341797 length of segment : 25 time for calcul the mask position with numpy : 0.00027108192443847656 nb_pixel_total : 6647 time to create 1 rle with old method : 0.013993263244628906 length of segment : 300 time for calcul the mask position with numpy : 9.34600830078125e-05 nb_pixel_total : 472 time to create 1 rle with old method : 0.001085519790649414 length of segment : 23 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -120.70391 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 17 time for calcul the mask position with numpy : 6.413459777832031e-05 nb_pixel_total : 650 time to create 1 rle with old method : 0.0008912086486816406 length of segment : 27 time for calcul the mask position with numpy : 3.3855438232421875e-05 nb_pixel_total : 115 time to create 1 rle with old method : 0.0001983642578125 length of segment : 12 time for calcul the mask position with numpy : 3.123283386230469e-05 nb_pixel_total : 157 time to create 1 rle with old method : 0.00024628639221191406 length of segment : 12 time for calcul the mask position with numpy : 3.4809112548828125e-05 nb_pixel_total : 309 time to create 1 rle with old method : 0.0005478858947753906 length of segment : 19 time for calcul the mask position with numpy : 8.749961853027344e-05 nb_pixel_total : 3078 time to create 1 rle with old method : 0.0038192272186279297 length of segment : 90 time for calcul the mask position with numpy : 4.076957702636719e-05 nb_pixel_total : 391 time to create 1 rle with old method : 0.0005848407745361328 length of segment : 32 time for calcul the mask position with numpy : 3.886222839355469e-05 nb_pixel_total : 31 time to create 1 rle with old method : 0.00016570091247558594 length of segment : 10 time for calcul the mask position with numpy : 0.0001304149627685547 nb_pixel_total : 7526 time to create 1 rle with old method : 0.00930643081665039 length of segment : 87 time for calcul the mask position with numpy : 5.340576171875e-05 nb_pixel_total : 872 time to create 1 rle with old method : 0.001222848892211914 length of segment : 31 time for calcul the mask position with numpy : 5.316734313964844e-05 nb_pixel_total : 793 time to create 1 rle with old method : 0.0011055469512939453 length of segment : 37 time for calcul the mask position with numpy : 3.790855407714844e-05 nb_pixel_total : 474 time to create 1 rle with old method : 0.0006802082061767578 length of segment : 27 time for calcul the mask position with numpy : 3.170967102050781e-05 nb_pixel_total : 159 time to create 1 rle with old method : 0.0002536773681640625 length of segment : 13 time for calcul the mask position with numpy : 3.3855438232421875e-05 nb_pixel_total : 187 time to create 1 rle with old method : 0.00028967857360839844 length of segment : 23 time for calcul the mask position with numpy : 0.0002052783966064453 nb_pixel_total : 1161 time to create 1 rle with old method : 0.0015544891357421875 length of segment : 63 time for calcul the mask position with numpy : 3.8623809814453125e-05 nb_pixel_total : 258 time to create 1 rle with old method : 0.0003819465637207031 length of segment : 16 time for calcul the mask position with numpy : 5.4836273193359375e-05 nb_pixel_total : 938 time to create 1 rle with old method : 0.0012714862823486328 length of segment : 131 time for calcul the mask position with numpy : 4.220008850097656e-05 nb_pixel_total : 409 time to create 1 rle with old method : 0.0005633831024169922 length of segment : 34 Processing 1 images image shape: (400, 400, 3) min: 2.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -113.82109 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 12 time for calcul the mask position with numpy : 8.630752563476562e-05 nb_pixel_total : 1133 time to create 1 rle with old method : 0.0018742084503173828 length of segment : 38 time for calcul the mask position with numpy : 0.00010538101196289062 nb_pixel_total : 1596 time to create 1 rle with old method : 0.002314329147338867 length of segment : 133 time for calcul the mask position with numpy : 9.298324584960938e-05 nb_pixel_total : 1737 time to create 1 rle with old method : 0.0025739669799804688 length of segment : 69 time for calcul the mask position with numpy : 6.198883056640625e-05 nb_pixel_total : 1130 time to create 1 rle with old method : 0.0017712116241455078 length of segment : 40 time for calcul the mask position with numpy : 7.128715515136719e-05 nb_pixel_total : 1324 time to create 1 rle with old method : 0.0018298625946044922 length of segment : 41 time for calcul the mask position with numpy : 4.9114227294921875e-05 nb_pixel_total : 211 time to create 1 rle with old method : 0.0004227161407470703 length of segment : 25 time for calcul the mask position with numpy : 5.7697296142578125e-05 nb_pixel_total : 1569 time to create 1 rle with old method : 0.002138853073120117 length of segment : 74 time for calcul the mask position with numpy : 6.079673767089844e-05 nb_pixel_total : 689 time to create 1 rle with old method : 0.0010988712310791016 length of segment : 24 time for calcul the mask position with numpy : 3.457069396972656e-05 nb_pixel_total : 160 time to create 1 rle with old method : 0.00025773048400878906 length of segment : 28 time for calcul the mask position with numpy : 6.079673767089844e-05 nb_pixel_total : 1901 time to create 1 rle with old method : 0.0025026798248291016 length of segment : 64 time for calcul the mask position with numpy : 6.175041198730469e-05 nb_pixel_total : 1818 time to create 1 rle with old method : 0.002292156219482422 length of segment : 84 time for calcul the mask position with numpy : 0.0002079010009765625 nb_pixel_total : 6782 time to create 1 rle with old method : 0.00910043716430664 length of segment : 210 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 127.51406 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 5 time for calcul the mask position with numpy : 0.0001049041748046875 nb_pixel_total : 1734 time to create 1 rle with old method : 0.0025053024291992188 length of segment : 84 time for calcul the mask position with numpy : 0.00011372566223144531 nb_pixel_total : 2013 time to create 1 rle with old method : 0.003232240676879883 length of segment : 120 time for calcul the mask position with numpy : 9.107589721679688e-05 nb_pixel_total : 1321 time to create 1 rle with old method : 0.0023469924926757812 length of segment : 105 time for calcul the mask position with numpy : 0.00016427040100097656 nb_pixel_total : 3750 time to create 1 rle with old method : 0.00493931770324707 length of segment : 196 time for calcul the mask position with numpy : 0.00021028518676757812 nb_pixel_total : 3208 time to create 1 rle with old method : 0.005522489547729492 length of segment : 174 Processing 1 images image shape: (280, 400, 3) min: 10.00000 max: 196.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 66.14297 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 1 time for calcul the mask position with numpy : 0.0013911724090576172 nb_pixel_total : 106467 time to create 1 rle with old method : 0.1595757007598877 length of segment : 281 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 130.95156 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 7 time for calcul the mask position with numpy : 7.081031799316406e-05 nb_pixel_total : 257 time to create 1 rle with old method : 0.0006241798400878906 length of segment : 28 time for calcul the mask position with numpy : 8.130073547363281e-05 nb_pixel_total : 118 time to create 1 rle with old method : 0.0002930164337158203 length of segment : 12 time for calcul the mask position with numpy : 0.00010347366333007812 nb_pixel_total : 2355 time to create 1 rle with old method : 0.005173206329345703 length of segment : 48 time for calcul the mask position with numpy : 4.1484832763671875e-05 nb_pixel_total : 183 time to create 1 rle with old method : 0.0003077983856201172 length of segment : 19 time for calcul the mask position with numpy : 5.555152893066406e-05 nb_pixel_total : 244 time to create 1 rle with old method : 0.0003752708435058594 length of segment : 17 time for calcul the mask position with numpy : 4.7206878662109375e-05 nb_pixel_total : 247 time to create 1 rle with old method : 0.0005211830139160156 length of segment : 17 time for calcul the mask position with numpy : 3.1948089599609375e-05 nb_pixel_total : 108 time to create 1 rle with old method : 0.0001800060272216797 length of segment : 12 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 142.33828 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 19 time for calcul the mask position with numpy : 6.079673767089844e-05 nb_pixel_total : 526 time to create 1 rle with old method : 0.0008089542388916016 length of segment : 20 time for calcul the mask position with numpy : 3.8623809814453125e-05 nb_pixel_total : 120 time to create 1 rle with old method : 0.00025391578674316406 length of segment : 18 time for calcul the mask position with numpy : 4.4345855712890625e-05 nb_pixel_total : 557 time to create 1 rle with old method : 0.0010957717895507812 length of segment : 30 time for calcul the mask position with numpy : 3.743171691894531e-05 nb_pixel_total : 201 time to create 1 rle with old method : 0.0003314018249511719 length of segment : 18 time for calcul the mask position with numpy : 4.76837158203125e-05 nb_pixel_total : 525 time to create 1 rle with old method : 0.0008058547973632812 length of segment : 25 time for calcul the mask position with numpy : 4.863739013671875e-05 nb_pixel_total : 358 time to create 1 rle with old method : 0.0005934238433837891 length of segment : 21 time for calcul the mask position with numpy : 4.267692565917969e-05 nb_pixel_total : 362 time to create 1 rle with old method : 0.0005347728729248047 length of segment : 34 time for calcul the mask position with numpy : 3.147125244140625e-05 nb_pixel_total : 111 time to create 1 rle with old method : 0.00018668174743652344 length of segment : 11 time for calcul the mask position with numpy : 3.4809112548828125e-05 nb_pixel_total : 203 time to create 1 rle with old method : 0.0003859996795654297 length of segment : 15 time for calcul the mask position with numpy : 3.9577484130859375e-05 nb_pixel_total : 128 time to create 1 rle with old method : 0.000217437744140625 length of segment : 16 time for calcul the mask position with numpy : 3.6716461181640625e-05 nb_pixel_total : 39 time to create 1 rle with old method : 8.678436279296875e-05 length of segment : 7 time for calcul the mask position with numpy : 4.482269287109375e-05 nb_pixel_total : 120 time to create 1 rle with old method : 0.00023674964904785156 length of segment : 14 time for calcul the mask position with numpy : 4.2438507080078125e-05 nb_pixel_total : 105 time to create 1 rle with old method : 0.00021386146545410156 length of segment : 10 time for calcul the mask position with numpy : 3.337860107421875e-05 nb_pixel_total : 243 time to create 1 rle with old method : 0.0003540515899658203 length of segment : 17 time for calcul the mask position with numpy : 4.887580871582031e-05 nb_pixel_total : 254 time to create 1 rle with old method : 0.00040793418884277344 length of segment : 20 time for calcul the mask position with numpy : 4.7206878662109375e-05 nb_pixel_total : 319 time to create 1 rle with old method : 0.000667572021484375 length of segment : 14 time for calcul the mask position with numpy : 6.0558319091796875e-05 nb_pixel_total : 495 time to create 1 rle with old method : 0.0007104873657226562 length of segment : 32 time for calcul the mask position with numpy : 4.410743713378906e-05 nb_pixel_total : 349 time to create 1 rle with old method : 0.0005705356597900391 length of segment : 25 time for calcul the mask position with numpy : 3.62396240234375e-05 nb_pixel_total : 339 time to create 1 rle with old method : 0.0005395412445068359 length of segment : 27 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 145.83437 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 5 time for calcul the mask position with numpy : 7.534027099609375e-05 nb_pixel_total : 137 time to create 1 rle with old method : 0.0004878044128417969 length of segment : 38 time for calcul the mask position with numpy : 4.887580871582031e-05 nb_pixel_total : 734 time to create 1 rle with old method : 0.0009477138519287109 length of segment : 72 time for calcul the mask position with numpy : 5.245208740234375e-05 nb_pixel_total : 412 time to create 1 rle with old method : 0.0006694793701171875 length of segment : 29 time for calcul the mask position with numpy : 3.600120544433594e-05 nb_pixel_total : 351 time to create 1 rle with old method : 0.0005435943603515625 length of segment : 24 time for calcul the mask position with numpy : 3.147125244140625e-05 nb_pixel_total : 109 time to create 1 rle with old method : 0.0001823902130126953 length of segment : 15 Processing 1 images image shape: (280, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 138.16250 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 2 time for calcul the mask position with numpy : 6.151199340820312e-05 nb_pixel_total : 17 time to create 1 rle with old method : 6.365776062011719e-05 length of segment : 12 time for calcul the mask position with numpy : 2.8848648071289062e-05 nb_pixel_total : 16 time to create 1 rle with old method : 4.458427429199219e-05 length of segment : 4 NEW PHOTO pour l'instant on ne peut pas sauvegarder la photo dans les tile Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -117.93438 max: 145.60391 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 4 time for calcul the mask position with numpy : 5.269050598144531e-05 nb_pixel_total : 1193 time to create 1 rle with old method : 0.0019371509552001953 length of segment : 31 time for calcul the mask position with numpy : 5.841255187988281e-05 nb_pixel_total : 918 time to create 1 rle with old method : 0.0012805461883544922 length of segment : 48 time for calcul the mask position with numpy : 7.891654968261719e-05 nb_pixel_total : 1650 time to create 1 rle with old method : 0.0024144649505615234 length of segment : 50 time for calcul the mask position with numpy : 7.653236389160156e-05 nb_pixel_total : 1337 time to create 1 rle with old method : 0.0019326210021972656 length of segment : 64 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -115.87969 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 11 time for calcul the mask position with numpy : 0.00015211105346679688 nb_pixel_total : 2779 time to create 1 rle with old method : 0.00621795654296875 length of segment : 98 time for calcul the mask position with numpy : 0.00010013580322265625 nb_pixel_total : 2005 time to create 1 rle with old method : 0.0037674903869628906 length of segment : 50 time for calcul the mask position with numpy : 4.315376281738281e-05 nb_pixel_total : 184 time to create 1 rle with old method : 0.0002758502960205078 length of segment : 19 time for calcul the mask position with numpy : 5.125999450683594e-05 nb_pixel_total : 967 time to create 1 rle with old method : 0.0015001296997070312 length of segment : 40 time for calcul the mask position with numpy : 4.482269287109375e-05 nb_pixel_total : 189 time to create 1 rle with old method : 0.00027251243591308594 length of segment : 29 time for calcul the mask position with numpy : 6.914138793945312e-05 nb_pixel_total : 3 time to create 1 rle with old method : 3.266334533691406e-05 length of segment : 3 time for calcul the mask position with numpy : 5.4836273193359375e-05 nb_pixel_total : 543 time to create 1 rle with old method : 0.0010190010070800781 length of segment : 37 time for calcul the mask position with numpy : 5.125999450683594e-05 nb_pixel_total : 557 time to create 1 rle with old method : 0.0007421970367431641 length of segment : 29 time for calcul the mask position with numpy : 7.510185241699219e-05 nb_pixel_total : 2649 time to create 1 rle with old method : 0.004027843475341797 length of segment : 52 time for calcul the mask position with numpy : 8.20159912109375e-05 nb_pixel_total : 2568 time to create 1 rle with old method : 0.0033926963806152344 length of segment : 50 time for calcul the mask position with numpy : 7.987022399902344e-05 nb_pixel_total : 2540 time to create 1 rle with old method : 0.0036497116088867188 length of segment : 53 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -103.44297 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 8 time for calcul the mask position with numpy : 6.699562072753906e-05 nb_pixel_total : 602 time to create 1 rle with old method : 0.0011000633239746094 length of segment : 45 time for calcul the mask position with numpy : 8.821487426757812e-05 nb_pixel_total : 1022 time to create 1 rle with old method : 0.002008199691772461 length of segment : 38 time for calcul the mask position with numpy : 0.00016951560974121094 nb_pixel_total : 6770 time to create 1 rle with old method : 0.012049674987792969 length of segment : 82 time for calcul the mask position with numpy : 5.793571472167969e-05 nb_pixel_total : 89 time to create 1 rle with old method : 0.00024318695068359375 length of segment : 9 time for calcul the mask position with numpy : 0.00014162063598632812 nb_pixel_total : 4290 time to create 1 rle with old method : 0.00786137580871582 length of segment : 93 time for calcul the mask position with numpy : 0.00017404556274414062 nb_pixel_total : 6946 time to create 1 rle with old method : 0.012762784957885742 length of segment : 90 time for calcul the mask position with numpy : 8.606910705566406e-05 nb_pixel_total : 722 time to create 1 rle with old method : 0.0012395381927490234 length of segment : 56 time for calcul the mask position with numpy : 0.00010919570922851562 nb_pixel_total : 2180 time to create 1 rle with old method : 0.003705263137817383 length of segment : 46 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -103.90000 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 4 time for calcul the mask position with numpy : 0.0001552104949951172 nb_pixel_total : 5846 time to create 1 rle with old method : 0.01036214828491211 length of segment : 93 time for calcul the mask position with numpy : 0.0001418590545654297 nb_pixel_total : 800 time to create 1 rle with old method : 0.0017256736755371094 length of segment : 45 time for calcul the mask position with numpy : 0.00010800361633300781 nb_pixel_total : 2151 time to create 1 rle with old method : 0.0027818679809570312 length of segment : 103 time for calcul the mask position with numpy : 6.556510925292969e-05 nb_pixel_total : 158 time to create 1 rle with old method : 0.00041103363037109375 length of segment : 24 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 5 time for calcul the mask position with numpy : 5.412101745605469e-05 nb_pixel_total : 511 time to create 1 rle with old method : 0.0010128021240234375 length of segment : 19 time for calcul the mask position with numpy : 7.200241088867188e-05 nb_pixel_total : 1534 time to create 1 rle with old method : 0.0026421546936035156 length of segment : 109 time for calcul the mask position with numpy : 5.9604644775390625e-05 nb_pixel_total : 1170 time to create 1 rle with old method : 0.005544185638427734 length of segment : 30 time for calcul the mask position with numpy : 4.363059997558594e-05 nb_pixel_total : 197 time to create 1 rle with old method : 0.00038361549377441406 length of segment : 29 time for calcul the mask position with numpy : 3.9577484130859375e-05 nb_pixel_total : 144 time to create 1 rle with old method : 0.0003247261047363281 length of segment : 14 Processing 1 images image shape: (400, 400, 3) min: 22.00000 max: 205.00000 molded_images shape: (1, 640, 640, 3) min: -94.80547 max: 76.93594 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 0 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -103.94609 max: 124.49844 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 4 time for calcul the mask position with numpy : 0.00016832351684570312 nb_pixel_total : 7724 time to create 1 rle with old method : 0.009085655212402344 length of segment : 153 time for calcul the mask position with numpy : 7.414817810058594e-05 nb_pixel_total : 527 time to create 1 rle with old method : 0.0013427734375 length of segment : 26 time for calcul the mask position with numpy : 3.8623809814453125e-05 nb_pixel_total : 402 time to create 1 rle with old method : 0.0005667209625244141 length of segment : 29 time for calcul the mask position with numpy : 0.00037407875061035156 nb_pixel_total : 25344 time to create 1 rle with old method : 0.03045511245727539 length of segment : 317 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -117.35234 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 11 time for calcul the mask position with numpy : 6.103515625e-05 nb_pixel_total : 654 time to create 1 rle with old method : 0.0012664794921875 length of segment : 29 time for calcul the mask position with numpy : 3.9577484130859375e-05 nb_pixel_total : 168 time to create 1 rle with old method : 0.00035643577575683594 length of segment : 12 time for calcul the mask position with numpy : 4.7206878662109375e-05 nb_pixel_total : 558 time to create 1 rle with old method : 0.0010380744934082031 length of segment : 40 time for calcul the mask position with numpy : 4.696846008300781e-05 nb_pixel_total : 294 time to create 1 rle with old method : 0.000598907470703125 length of segment : 17 time for calcul the mask position with numpy : 9.1552734375e-05 nb_pixel_total : 2649 time to create 1 rle with old method : 0.004913330078125 length of segment : 43 time for calcul the mask position with numpy : 0.00019121170043945312 nb_pixel_total : 4377 time to create 1 rle with old method : 0.007950067520141602 length of segment : 105 time for calcul the mask position with numpy : 0.00010848045349121094 nb_pixel_total : 2550 time to create 1 rle with old method : 0.0033044815063476562 length of segment : 88 time for calcul the mask position with numpy : 4.863739013671875e-05 nb_pixel_total : 195 time to create 1 rle with old method : 0.0003733634948730469 length of segment : 25 time for calcul the mask position with numpy : 0.00017786026000976562 nb_pixel_total : 8490 time to create 1 rle with old method : 0.01248788833618164 length of segment : 140 time for calcul the mask position with numpy : 7.796287536621094e-05 nb_pixel_total : 2427 time to create 1 rle with old method : 0.0033042430877685547 length of segment : 39 time for calcul the mask position with numpy : 7.200241088867188e-05 nb_pixel_total : 2377 time to create 1 rle with old method : 0.0031142234802246094 length of segment : 38 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -117.26641 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 8 time for calcul the mask position with numpy : 0.0001239776611328125 nb_pixel_total : 1562 time to create 1 rle with old method : 0.0020515918731689453 length of segment : 126 time for calcul the mask position with numpy : 3.9577484130859375e-05 nb_pixel_total : 325 time to create 1 rle with old method : 0.00047707557678222656 length of segment : 29 time for calcul the mask position with numpy : 5.459785461425781e-05 nb_pixel_total : 1255 time to create 1 rle with old method : 0.0016543865203857422 length of segment : 39 time for calcul the mask position with numpy : 0.00010037422180175781 nb_pixel_total : 1404 time to create 1 rle with old method : 0.0018618106842041016 length of segment : 118 time for calcul the mask position with numpy : 4.9114227294921875e-05 nb_pixel_total : 729 time to create 1 rle with old method : 0.001081228256225586 length of segment : 21 time for calcul the mask position with numpy : 7.915496826171875e-05 nb_pixel_total : 1638 time to create 1 rle with old method : 0.002159595489501953 length of segment : 88 time for calcul the mask position with numpy : 9.226799011230469e-05 nb_pixel_total : 3377 time to create 1 rle with old method : 0.005054473876953125 length of segment : 87 time for calcul the mask position with numpy : 4.863739013671875e-05 nb_pixel_total : 1427 time to create 1 rle with old method : 0.0018436908721923828 length of segment : 44 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 126.56875 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 6 time for calcul the mask position with numpy : 5.316734313964844e-05 nb_pixel_total : 931 time to create 1 rle with old method : 0.0012383460998535156 length of segment : 35 time for calcul the mask position with numpy : 7.05718994140625e-05 nb_pixel_total : 598 time to create 1 rle with old method : 0.0008134841918945312 length of segment : 58 time for calcul the mask position with numpy : 5.6743621826171875e-05 nb_pixel_total : 724 time to create 1 rle with old method : 0.0009796619415283203 length of segment : 79 time for calcul the mask position with numpy : 6.818771362304688e-05 nb_pixel_total : 1631 time to create 1 rle with old method : 0.0020842552185058594 length of segment : 86 time for calcul the mask position with numpy : 5.078315734863281e-05 nb_pixel_total : 335 time to create 1 rle with old method : 0.0005168914794921875 length of segment : 25 time for calcul the mask position with numpy : 4.9114227294921875e-05 nb_pixel_total : 959 time to create 1 rle with old method : 0.0012824535369873047 length of segment : 52 Processing 1 images image shape: (280, 400, 3) min: 7.00000 max: 191.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 65.38516 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 2 time for calcul the mask position with numpy : 3.8623809814453125e-05 nb_pixel_total : 180 time to create 1 rle with old method : 0.00026726722717285156 length of segment : 19 time for calcul the mask position with numpy : 0.0009500980377197266 nb_pixel_total : 106117 time to create 1 rle with old method : 0.1498546600341797 length of segment : 280 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 128.71719 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 12 time for calcul the mask position with numpy : 4.267692565917969e-05 nb_pixel_total : 134 time to create 1 rle with old method : 0.00021386146545410156 length of segment : 13 time for calcul the mask position with numpy : 6.079673767089844e-05 nb_pixel_total : 261 time to create 1 rle with old method : 0.0007548332214355469 length of segment : 18 time for calcul the mask position with numpy : 3.6716461181640625e-05 nb_pixel_total : 134 time to create 1 rle with old method : 0.00022172927856445312 length of segment : 12 time for calcul the mask position with numpy : 3.123283386230469e-05 nb_pixel_total : 56 time to create 1 rle with old method : 0.0001316070556640625 length of segment : 12 time for calcul the mask position with numpy : 3.719329833984375e-05 nb_pixel_total : 405 time to create 1 rle with old method : 0.0005276203155517578 length of segment : 33 time for calcul the mask position with numpy : 0.00010824203491210938 nb_pixel_total : 4703 time to create 1 rle with old method : 0.0060634613037109375 length of segment : 109 time for calcul the mask position with numpy : 6.961822509765625e-05 nb_pixel_total : 970 time to create 1 rle with old method : 0.0013649463653564453 length of segment : 51 time for calcul the mask position with numpy : 4.1961669921875e-05 nb_pixel_total : 236 time to create 1 rle with old method : 0.00035071372985839844 length of segment : 20 time for calcul the mask position with numpy : 3.361701965332031e-05 nb_pixel_total : 86 time to create 1 rle with old method : 0.00017690658569335938 length of segment : 13 time for calcul the mask position with numpy : 5.14984130859375e-05 nb_pixel_total : 892 time to create 1 rle with old method : 0.0012679100036621094 length of segment : 29 time for calcul the mask position with numpy : 3.695487976074219e-05 nb_pixel_total : 219 time to create 1 rle with old method : 0.00036525726318359375 length of segment : 16 time for calcul the mask position with numpy : 0.00010466575622558594 nb_pixel_total : 4449 time to create 1 rle with old method : 0.005706071853637695 length of segment : 91 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 140.88125 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 21 time for calcul the mask position with numpy : 6.318092346191406e-05 nb_pixel_total : 633 time to create 1 rle with old method : 0.0009267330169677734 length of segment : 32 time for calcul the mask position with numpy : 3.504753112792969e-05 nb_pixel_total : 114 time to create 1 rle with old method : 0.00020194053649902344 length of segment : 13 time for calcul the mask position with numpy : 3.647804260253906e-05 nb_pixel_total : 235 time to create 1 rle with old method : 0.0004105567932128906 length of segment : 18 time for calcul the mask position with numpy : 3.886222839355469e-05 nb_pixel_total : 347 time to create 1 rle with old method : 0.0004978179931640625 length of segment : 19 time for calcul the mask position with numpy : 3.2901763916015625e-05 nb_pixel_total : 113 time to create 1 rle with old method : 0.00018167495727539062 length of segment : 18 time for calcul the mask position with numpy : 3.600120544433594e-05 nb_pixel_total : 363 time to create 1 rle with old method : 0.00048279762268066406 length of segment : 31 time for calcul the mask position with numpy : 6.29425048828125e-05 nb_pixel_total : 543 time to create 1 rle with old method : 0.000823974609375 length of segment : 22 time for calcul the mask position with numpy : 7.343292236328125e-05 nb_pixel_total : 179 time to create 1 rle with old method : 0.0005185604095458984 length of segment : 14 time for calcul the mask position with numpy : 4.649162292480469e-05 nb_pixel_total : 123 time to create 1 rle with old method : 0.00030994415283203125 length of segment : 12 time for calcul the mask position with numpy : 5.698204040527344e-05 nb_pixel_total : 571 time to create 1 rle with old method : 0.00095367431640625 length of segment : 24 time for calcul the mask position with numpy : 4.38690185546875e-05 nb_pixel_total : 327 time to create 1 rle with old method : 0.00044465065002441406 length of segment : 25 time for calcul the mask position with numpy : 3.24249267578125e-05 nb_pixel_total : 105 time to create 1 rle with old method : 0.00024318695068359375 length of segment : 12 time for calcul the mask position with numpy : 3.719329833984375e-05 nb_pixel_total : 103 time to create 1 rle with old method : 0.0001704692840576172 length of segment : 13 time for calcul the mask position with numpy : 3.504753112792969e-05 nb_pixel_total : 355 time to create 1 rle with old method : 0.0004870891571044922 length of segment : 27 time for calcul the mask position with numpy : 3.4332275390625e-05 nb_pixel_total : 150 time to create 1 rle with old method : 0.00022530555725097656 length of segment : 17 time for calcul the mask position with numpy : 3.600120544433594e-05 nb_pixel_total : 223 time to create 1 rle with old method : 0.0003476142883300781 length of segment : 17 time for calcul the mask position with numpy : 4.100799560546875e-05 nb_pixel_total : 203 time to create 1 rle with old method : 0.0003795623779296875 length of segment : 18 time for calcul the mask position with numpy : 4.410743713378906e-05 nb_pixel_total : 374 time to create 1 rle with old method : 0.0006325244903564453 length of segment : 28 time for calcul the mask position with numpy : 4.00543212890625e-05 nb_pixel_total : 218 time to create 1 rle with old method : 0.0004298686981201172 length of segment : 12 time for calcul the mask position with numpy : 4.172325134277344e-05 nb_pixel_total : 277 time to create 1 rle with old method : 0.0005009174346923828 length of segment : 19 time for calcul the mask position with numpy : 3.218650817871094e-05 nb_pixel_total : 194 time to create 1 rle with old method : 0.000293731689453125 length of segment : 19 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 145.56484 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 6 time for calcul the mask position with numpy : 5.626678466796875e-05 nb_pixel_total : 420 time to create 1 rle with old method : 0.0006520748138427734 length of segment : 62 time for calcul the mask position with numpy : 4.887580871582031e-05 nb_pixel_total : 289 time to create 1 rle with old method : 0.00044655799865722656 length of segment : 61 time for calcul the mask position with numpy : 4.482269287109375e-05 nb_pixel_total : 83 time to create 1 rle with old method : 0.00019979476928710938 length of segment : 9 time for calcul the mask position with numpy : 3.147125244140625e-05 nb_pixel_total : 121 time to create 1 rle with old method : 0.00019049644470214844 length of segment : 18 time for calcul the mask position with numpy : 3.6716461181640625e-05 nb_pixel_total : 372 time to create 1 rle with old method : 0.0005297660827636719 length of segment : 27 time for calcul the mask position with numpy : 3.695487976074219e-05 nb_pixel_total : 405 time to create 1 rle with old method : 0.0005540847778320312 length of segment : 35 Processing 1 images image shape: (280, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 140.28750 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 2 time for calcul the mask position with numpy : 9.298324584960938e-05 nb_pixel_total : 56 time to create 1 rle with old method : 0.00031185150146484375 length of segment : 27 time for calcul the mask position with numpy : 3.695487976074219e-05 nb_pixel_total : 15 time to create 1 rle with old method : 5.91278076171875e-05 length of segment : 3 Detection mask done ! Trying to reset tf kernel 494952 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5393 tf kernel not reseted sub process len(results) : 199 len(list_Values) 199 None max_time_sub_proc : 3600 parent process len(results) : 0 len(list_Values) 199 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 10274 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Catched exception ! Connect or reconnect ! Number saved : None batch 1 Loaded 199 chid ids of type : 4228 Number RLEs to save : 9704 save missing photos in datou_result : time spend for datou_step_exec : 26.106095552444458 time spend to save output : 0.8563377857208252 total time spend for step 1 : 26.962433338165283 step2:brightness Tue Feb 25 11:20:55 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure inside step calcul brightness treat image : temp/1740478828_494175_1339526600_d41129f889416a13d07ad3fbef947a6e.jpg treat image : temp/1740478828_494175_1339526597_1fd5da1c85e708a6dd60a56c182ca406.jpg Inside saveOutput : final : False verbose : 0 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 2 time used for this insertion : 0.011810779571533203 begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 2 time used for this insertion : 0.011978387832641602 save missing photos in datou_result : time spend for datou_step_exec : 1.0213050842285156 time spend to save output : 0.028548002243041992 total time spend for step 2 : 1.0498530864715576 step3:blur_detection Tue Feb 25 11:20:56 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure inside step blur_detection methode: ratio et variance treat image : temp/1740478828_494175_1339526600_d41129f889416a13d07ad3fbef947a6e.jpg resize: (1080, 1920) 1339526600 -7.134587080315985 treat image : temp/1740478828_494175_1339526597_1fd5da1c85e708a6dd60a56c182ca406.jpg resize: (1080, 1920) 1339526597 -7.0980378304370095 Inside saveOutput : final : False verbose : 0 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 2 time used for this insertion : 0.01377248764038086 begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 2 time used for this insertion : 0.01357412338256836 save missing photos in datou_result : time spend for datou_step_exec : 2.1775505542755127 time spend to save output : 0.032523155212402344 total time spend for step 3 : 2.210073709487915 step4:rle_unique_nms_with_priority Tue Feb 25 11:20:59 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms batch 1 Loaded 199 chid ids of type : 4228 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++nb_obj : 95 nb_hashtags : 8 time to prepare the origin masks : 1.0862929821014404 time for calcul the mask position with numpy : 0.761127233505249 nb_pixel_total : 1873542 time to create 1 rle with new method : 0.16346311569213867 time for calcul the mask position with numpy : 0.008677244186401367 nb_pixel_total : 1482 time to create 1 rle with old method : 0.0017616748809814453 time for calcul the mask position with numpy : 0.008741617202758789 nb_pixel_total : 1435 time to create 1 rle with old method : 0.0016605854034423828 time for calcul the mask position with numpy : 0.008758068084716797 nb_pixel_total : 253 time to create 1 rle with old method : 0.00033855438232421875 time for calcul the mask position with numpy : 0.00874471664428711 nb_pixel_total : 751 time to create 1 rle with old method : 0.0008788108825683594 time for calcul the mask position with numpy : 0.008765697479248047 nb_pixel_total : 1079 time to create 1 rle with old method : 0.0012309551239013672 time for calcul the mask position with numpy : 0.008730888366699219 nb_pixel_total : 1241 time to create 1 rle with old method : 0.0013926029205322266 time for calcul the mask position with numpy : 0.008593082427978516 nb_pixel_total : 1066 time to create 1 rle with old method : 0.0012257099151611328 time for calcul the mask position with numpy : 0.008675336837768555 nb_pixel_total : 698 time to create 1 rle with old method : 0.0008022785186767578 time for calcul the mask position with numpy : 0.010404586791992188 nb_pixel_total : 2377 time to create 1 rle with old method : 0.0034596920013427734 time for calcul the mask position with numpy : 0.008748292922973633 nb_pixel_total : 99 time to create 1 rle with old method : 0.0001800060272216797 time for calcul the mask position with numpy : 0.008695840835571289 nb_pixel_total : 677 time to create 1 rle with old method : 0.0007762908935546875 time for calcul the mask position with numpy : 0.008589029312133789 nb_pixel_total : 540 time to create 1 rle with old method : 0.0006339550018310547 time for calcul the mask position with numpy : 0.008913755416870117 nb_pixel_total : 2332 time to create 1 rle with old method : 0.0029175281524658203 time for calcul the mask position with numpy : 0.009735345840454102 nb_pixel_total : 932 time to create 1 rle with old method : 0.001489877700805664 time for calcul the mask position with numpy : 0.008521080017089844 nb_pixel_total : 1977 time to create 1 rle with old method : 0.0022766590118408203 time for calcul the mask position with numpy : 0.009124279022216797 nb_pixel_total : 28 time to create 1 rle with old method : 0.00011134147644042969 time for calcul the mask position with numpy : 0.009165763854980469 nb_pixel_total : 36 time to create 1 rle with old method : 0.00012874603271484375 time for calcul the mask position with numpy : 0.008916854858398438 nb_pixel_total : 5722 time to create 1 rle with old method : 0.006655216217041016 time for calcul the mask position with numpy : 0.008629083633422852 nb_pixel_total : 5964 time to create 1 rle with old method : 0.006752729415893555 time for calcul the mask position with numpy : 0.009111642837524414 nb_pixel_total : 4 time to create 1 rle with old method : 3.647804260253906e-05 time for calcul the mask position with numpy : 0.008603334426879883 nb_pixel_total : 202 time to create 1 rle with old method : 0.0002510547637939453 time for calcul the mask position with numpy : 0.008821964263916016 nb_pixel_total : 2419 time to create 1 rle with old method : 0.002753734588623047 time for calcul the mask position with numpy : 0.009373903274536133 nb_pixel_total : 3202 time to create 1 rle with old method : 0.005967855453491211 time for calcul the mask position with numpy : 0.011722087860107422 nb_pixel_total : 514 time to create 1 rle with old method : 0.0008797645568847656 time for calcul the mask position with numpy : 0.011587858200073242 nb_pixel_total : 309 time to create 1 rle with old method : 0.0005605220794677734 time for calcul the mask position with numpy : 0.010202646255493164 nb_pixel_total : 7362 time to create 1 rle with old method : 0.011972427368164062 time for calcul the mask position with numpy : 0.010018587112426758 nb_pixel_total : 798 time to create 1 rle with old method : 0.0014057159423828125 time for calcul the mask position with numpy : 0.010064363479614258 nb_pixel_total : 6782 time to create 1 rle with old method : 0.007834196090698242 time for calcul the mask position with numpy : 0.008599281311035156 nb_pixel_total : 391 time to create 1 rle with old method : 0.00046753883361816406 time for calcul the mask position with numpy : 0.008525848388671875 nb_pixel_total : 10 time to create 1 rle with old method : 3.457069396972656e-05 time for calcul the mask position with numpy : 0.008578777313232422 nb_pixel_total : 474 time to create 1 rle with old method : 0.0005793571472167969 time for calcul the mask position with numpy : 0.008478164672851562 nb_pixel_total : 305 time to create 1 rle with old method : 0.00041747093200683594 time for calcul the mask position with numpy : 0.008538246154785156 nb_pixel_total : 1569 time to create 1 rle with old method : 0.0018754005432128906 time for calcul the mask position with numpy : 0.008479118347167969 nb_pixel_total : 1161 time to create 1 rle with old method : 0.0013811588287353516 time for calcul the mask position with numpy : 0.008508682250976562 nb_pixel_total : 1737 time to create 1 rle with old method : 0.002001047134399414 time for calcul the mask position with numpy : 0.008456707000732422 nb_pixel_total : 3078 time to create 1 rle with old method : 0.003468751907348633 time for calcul the mask position with numpy : 0.008475303649902344 nb_pixel_total : 207 time to create 1 rle with old method : 0.00026798248291015625 time for calcul the mask position with numpy : 0.008533716201782227 nb_pixel_total : 31 time to create 1 rle with old method : 8.869171142578125e-05 time for calcul the mask position with numpy : 0.008448123931884766 nb_pixel_total : 1324 time to create 1 rle with old method : 0.0014867782592773438 time for calcul the mask position with numpy : 0.008597850799560547 nb_pixel_total : 211 time to create 1 rle with old method : 0.00026035308837890625 time for calcul the mask position with numpy : 0.008465766906738281 nb_pixel_total : 6457 time to create 1 rle with old method : 0.0074291229248046875 time for calcul the mask position with numpy : 0.00855398178100586 nb_pixel_total : 2013 time to create 1 rle with old method : 0.0023479461669921875 time for calcul the mask position with numpy : 0.008566617965698242 nb_pixel_total : 55 time to create 1 rle with old method : 0.0001342296600341797 time for calcul the mask position with numpy : 0.008470773696899414 nb_pixel_total : 3750 time to create 1 rle with old method : 0.004382610321044922 time for calcul the mask position with numpy : 0.008481025695800781 nb_pixel_total : 1596 time to create 1 rle with old method : 0.0018792152404785156 time for calcul the mask position with numpy : 0.008475780487060547 nb_pixel_total : 157 time to create 1 rle with old method : 0.0002529621124267578 time for calcul the mask position with numpy : 0.008580207824707031 nb_pixel_total : 325 time to create 1 rle with old method : 0.00042939186096191406 time for calcul the mask position with numpy : 0.008506536483764648 nb_pixel_total : 872 time to create 1 rle with old method : 0.0010099411010742188 time for calcul the mask position with numpy : 0.008502006530761719 nb_pixel_total : 1130 time to create 1 rle with old method : 0.0013186931610107422 time for calcul the mask position with numpy : 0.008629560470581055 nb_pixel_total : 157 time to create 1 rle with old method : 0.00020003318786621094 time for calcul the mask position with numpy : 0.008515596389770508 nb_pixel_total : 160 time to create 1 rle with old method : 0.0002052783966064453 time for calcul the mask position with numpy : 0.008947372436523438 nb_pixel_total : 16 time to create 1 rle with old method : 4.1961669921875e-05 time for calcul the mask position with numpy : 0.008565902709960938 nb_pixel_total : 1734 time to create 1 rle with old method : 0.002007722854614258 time for calcul the mask position with numpy : 0.008513927459716797 nb_pixel_total : 1133 time to create 1 rle with old method : 0.0013306140899658203 time for calcul the mask position with numpy : 0.008606910705566406 nb_pixel_total : 472 time to create 1 rle with old method : 0.0005540847778320312 time for calcul the mask position with numpy : 0.008515119552612305 nb_pixel_total : 258 time to create 1 rle with old method : 0.0003325939178466797 time for calcul the mask position with numpy : 0.008433341979980469 nb_pixel_total : 115 time to create 1 rle with old method : 0.000152587890625 time for calcul the mask position with numpy : 0.008515357971191406 nb_pixel_total : 650 time to create 1 rle with old method : 0.0007543563842773438 time for calcul the mask position with numpy : 0.00847768783569336 nb_pixel_total : 73 time to create 1 rle with old method : 0.00010395050048828125 time for calcul the mask position with numpy : 0.008548736572265625 nb_pixel_total : 793 time to create 1 rle with old method : 0.0009417533874511719 time for calcul the mask position with numpy : 0.008474588394165039 nb_pixel_total : 689 time to create 1 rle with old method : 0.0008175373077392578 time for calcul the mask position with numpy : 0.009412527084350586 nb_pixel_total : 106467 time to create 1 rle with old method : 0.12432718276977539 time for calcul the mask position with numpy : 0.008625984191894531 nb_pixel_total : 118 time to create 1 rle with old method : 0.0001747608184814453 time for calcul the mask position with numpy : 0.008532524108886719 nb_pixel_total : 2 time to create 1 rle with old method : 1.7881393432617188e-05 time for calcul the mask position with numpy : 0.008477449417114258 nb_pixel_total : 17 time to create 1 rle with old method : 3.600120544433594e-05 time for calcul the mask position with numpy : 0.008446216583251953 nb_pixel_total : 247 time to create 1 rle with old method : 0.0003192424774169922 time for calcul the mask position with numpy : 0.0084381103515625 nb_pixel_total : 257 time to create 1 rle with old method : 0.0003077983856201172 time for calcul the mask position with numpy : 0.008445024490356445 nb_pixel_total : 10 time to create 1 rle with old method : 2.9325485229492188e-05 time for calcul the mask position with numpy : 0.008448123931884766 nb_pixel_total : 349 time to create 1 rle with old method : 0.0004200935363769531 time for calcul the mask position with numpy : 0.008436441421508789 nb_pixel_total : 734 time to create 1 rle with old method : 0.0008664131164550781 time for calcul the mask position with numpy : 0.008446455001831055 nb_pixel_total : 203 time to create 1 rle with old method : 0.00023937225341796875 time for calcul the mask position with numpy : 0.00843048095703125 nb_pixel_total : 525 time to create 1 rle with old method : 0.0006191730499267578 time for calcul the mask position with numpy : 0.008464813232421875 nb_pixel_total : 358 time to create 1 rle with old method : 0.00043129920959472656 time for calcul the mask position with numpy : 0.008457660675048828 nb_pixel_total : 557 time to create 1 rle with old method : 0.0006611347198486328 time for calcul the mask position with numpy : 0.00847005844116211 nb_pixel_total : 183 time to create 1 rle with old method : 0.0002276897430419922 time for calcul the mask position with numpy : 0.008428573608398438 nb_pixel_total : 2355 time to create 1 rle with old method : 0.002749919891357422 time for calcul the mask position with numpy : 0.00849461555480957 nb_pixel_total : 111 time to create 1 rle with old method : 0.0001423358917236328 time for calcul the mask position with numpy : 0.008490324020385742 nb_pixel_total : 244 time to create 1 rle with old method : 0.0002932548522949219 time for calcul the mask position with numpy : 0.008470535278320312 nb_pixel_total : 412 time to create 1 rle with old method : 0.0004885196685791016 time for calcul the mask position with numpy : 0.008484363555908203 nb_pixel_total : 362 time to create 1 rle with old method : 0.0004394054412841797 time for calcul the mask position with numpy : 0.00864720344543457 nb_pixel_total : 243 time to create 1 rle with old method : 0.0002944469451904297 time for calcul the mask position with numpy : 0.009369373321533203 nb_pixel_total : 128 time to create 1 rle with old method : 0.00021648406982421875 time for calcul the mask position with numpy : 0.008917570114135742 nb_pixel_total : 319 time to create 1 rle with old method : 0.0003924369812011719 time for calcul the mask position with numpy : 0.008506298065185547 nb_pixel_total : 109 time to create 1 rle with old method : 0.00015211105346679688 time for calcul the mask position with numpy : 0.00848698616027832 nb_pixel_total : 526 time to create 1 rle with old method : 0.0006096363067626953 time for calcul the mask position with numpy : 0.008547544479370117 nb_pixel_total : 137 time to create 1 rle with old method : 0.0002117156982421875 time for calcul the mask position with numpy : 0.008613109588623047 nb_pixel_total : 201 time to create 1 rle with old method : 0.00024271011352539062 time for calcul the mask position with numpy : 0.008577823638916016 nb_pixel_total : 16 time to create 1 rle with old method : 4.982948303222656e-05 time for calcul the mask position with numpy : 0.008527755737304688 nb_pixel_total : 495 time to create 1 rle with old method : 0.000576019287109375 time for calcul the mask position with numpy : 0.00854039192199707 nb_pixel_total : 120 time to create 1 rle with old method : 0.00016355514526367188 time for calcul the mask position with numpy : 0.008527040481567383 nb_pixel_total : 254 time to create 1 rle with old method : 0.0003085136413574219 time for calcul the mask position with numpy : 0.008429765701293945 nb_pixel_total : 120 time to create 1 rle with old method : 0.00016760826110839844 time for calcul the mask position with numpy : 0.008549213409423828 nb_pixel_total : 39 time to create 1 rle with old method : 6.222724914550781e-05 time for calcul the mask position with numpy : 0.008501529693603516 nb_pixel_total : 105 time to create 1 rle with old method : 0.0001418590545654297 time for calcul the mask position with numpy : 0.008464574813842773 nb_pixel_total : 351 time to create 1 rle with old method : 0.0004215240478515625 create new chi : 2.0118257999420166 after preparing all the mask , begin to delete the rle from the crop_hashtag_id => VR 28-11-20 : il faut déplacer cela apres le save_crop_hashtag_ids_obj de la ligne 9514 ! we have 0 chi objets contains the rles time to delete rle : 0.006252288818359375 batch 1 Loaded 96 chid ids of type : 4230 Number RLEs to save : 9559 TO DO : save crop sub photo not yet done ! save time : 0.5953502655029297 nb_obj : 103 nb_hashtags : 7 time to prepare the origin masks : 2.1282947063446045 time for calcul the mask position with numpy : 0.32360076904296875 nb_pixel_total : 1842097 time to create 1 rle with new method : 0.6556351184844971 time for calcul the mask position with numpy : 0.013492822647094727 nb_pixel_total : 2180 time to create 1 rle with old method : 0.0025970935821533203 time for calcul the mask position with numpy : 0.011799812316894531 nb_pixel_total : 1534 time to create 1 rle with old method : 0.0023648738861083984 time for calcul the mask position with numpy : 0.011811971664428711 nb_pixel_total : 189 time to create 1 rle with old method : 0.0002732276916503906 time for calcul the mask position with numpy : 0.010248422622680664 nb_pixel_total : 602 time to create 1 rle with old method : 0.0007915496826171875 time for calcul the mask position with numpy : 0.010785818099975586 nb_pixel_total : 967 time to create 1 rle with old method : 0.0011510848999023438 time for calcul the mask position with numpy : 0.010855674743652344 nb_pixel_total : 18 time to create 1 rle with old method : 6.771087646484375e-05 time for calcul the mask position with numpy : 0.011155843734741211 nb_pixel_total : 543 time to create 1 rle with old method : 0.0006453990936279297 time for calcul the mask position with numpy : 0.009513616561889648 nb_pixel_total : 918 time to create 1 rle with old method : 0.0015213489532470703 time for calcul the mask position with numpy : 0.012130498886108398 nb_pixel_total : 1193 time to create 1 rle with old method : 0.001425027847290039 time for calcul the mask position with numpy : 0.011395931243896484 nb_pixel_total : 147 time to create 1 rle with old method : 0.0002276897430419922 time for calcul the mask position with numpy : 0.010816335678100586 nb_pixel_total : 1650 time to create 1 rle with old method : 0.001984834671020508 time for calcul the mask position with numpy : 0.008316516876220703 nb_pixel_total : 3 time to create 1 rle with old method : 3.409385681152344e-05 time for calcul the mask position with numpy : 0.008721590042114258 nb_pixel_total : 2649 time to create 1 rle with old method : 0.003567218780517578 time for calcul the mask position with numpy : 0.0079803466796875 nb_pixel_total : 174 time to create 1 rle with old method : 0.0003685951232910156 time for calcul the mask position with numpy : 0.007312297821044922 nb_pixel_total : 36 time to create 1 rle with old method : 0.00013947486877441406 time for calcul the mask position with numpy : 0.007114410400390625 nb_pixel_total : 158 time to create 1 rle with old method : 0.00020885467529296875 time for calcul the mask position with numpy : 0.007302761077880859 nb_pixel_total : 197 time to create 1 rle with old method : 0.00026106834411621094 time for calcul the mask position with numpy : 0.007291078567504883 nb_pixel_total : 2779 time to create 1 rle with old method : 0.0037386417388916016 time for calcul the mask position with numpy : 0.011959552764892578 nb_pixel_total : 2151 time to create 1 rle with old method : 0.0034961700439453125 time for calcul the mask position with numpy : 0.012119293212890625 nb_pixel_total : 1022 time to create 1 rle with old method : 0.0013248920440673828 time for calcul the mask position with numpy : 0.011420249938964844 nb_pixel_total : 800 time to create 1 rle with old method : 0.00099945068359375 time for calcul the mask position with numpy : 0.011567354202270508 nb_pixel_total : 2005 time to create 1 rle with old method : 0.002507448196411133 time for calcul the mask position with numpy : 0.011623382568359375 nb_pixel_total : 722 time to create 1 rle with old method : 0.0010600090026855469 time for calcul the mask position with numpy : 0.012452125549316406 nb_pixel_total : 5846 time to create 1 rle with old method : 0.007189035415649414 time for calcul the mask position with numpy : 0.011655092239379883 nb_pixel_total : 4290 time to create 1 rle with old method : 0.005146980285644531 time for calcul the mask position with numpy : 0.012390851974487305 nb_pixel_total : 144 time to create 1 rle with old method : 0.0002913475036621094 time for calcul the mask position with numpy : 0.01132345199584961 nb_pixel_total : 89 time to create 1 rle with old method : 0.00013756752014160156 time for calcul the mask position with numpy : 0.011182308197021484 nb_pixel_total : 184 time to create 1 rle with old method : 0.00024247169494628906 time for calcul the mask position with numpy : 0.011672496795654297 nb_pixel_total : 658 time to create 1 rle with old method : 0.00087738037109375 time for calcul the mask position with numpy : 0.01375269889831543 nb_pixel_total : 1170 time to create 1 rle with old method : 0.00138092041015625 time for calcul the mask position with numpy : 0.011476278305053711 nb_pixel_total : 6770 time to create 1 rle with old method : 0.00806117057800293 time for calcul the mask position with numpy : 0.011554241180419922 nb_pixel_total : 511 time to create 1 rle with old method : 0.0006396770477294922 time for calcul the mask position with numpy : 0.007844209671020508 nb_pixel_total : 4 time to create 1 rle with old method : 3.7670135498046875e-05 time for calcul the mask position with numpy : 0.007659435272216797 nb_pixel_total : 2649 time to create 1 rle with old method : 0.0030155181884765625 time for calcul the mask position with numpy : 0.007182121276855469 nb_pixel_total : 558 time to create 1 rle with old method : 0.0007088184356689453 time for calcul the mask position with numpy : 0.007146358489990234 nb_pixel_total : 1562 time to create 1 rle with old method : 0.001912832260131836 time for calcul the mask position with numpy : 0.00722050666809082 nb_pixel_total : 4377 time to create 1 rle with old method : 0.0051114559173583984 time for calcul the mask position with numpy : 0.007221698760986328 nb_pixel_total : 25344 time to create 1 rle with old method : 0.03004145622253418 time for calcul the mask position with numpy : 0.0072133541107177734 nb_pixel_total : 8490 time to create 1 rle with old method : 0.010084867477416992 time for calcul the mask position with numpy : 0.007082939147949219 nb_pixel_total : 1893 time to create 1 rle with old method : 0.0023298263549804688 time for calcul the mask position with numpy : 0.009274005889892578 nb_pixel_total : 1638 time to create 1 rle with old method : 0.0021126270294189453 time for calcul the mask position with numpy : 0.00787663459777832 nb_pixel_total : 2550 time to create 1 rle with old method : 0.003693103790283203 time for calcul the mask position with numpy : 0.007672548294067383 nb_pixel_total : 182 time to create 1 rle with old method : 0.0002772808074951172 time for calcul the mask position with numpy : 0.007858991622924805 nb_pixel_total : 1255 time to create 1 rle with old method : 0.0014166831970214844 time for calcul the mask position with numpy : 0.00796961784362793 nb_pixel_total : 325 time to create 1 rle with old method : 0.0004031658172607422 time for calcul the mask position with numpy : 0.008347034454345703 nb_pixel_total : 959 time to create 1 rle with old method : 0.0012073516845703125 time for calcul the mask position with numpy : 0.007558584213256836 nb_pixel_total : 7724 time to create 1 rle with old method : 0.009240865707397461 time for calcul the mask position with numpy : 0.006991386413574219 nb_pixel_total : 931 time to create 1 rle with old method : 0.001068115234375 time for calcul the mask position with numpy : 0.007032632827758789 nb_pixel_total : 598 time to create 1 rle with old method : 0.0007596015930175781 time for calcul the mask position with numpy : 0.007095813751220703 nb_pixel_total : 300 time to create 1 rle with old method : 0.00037670135498046875 time for calcul the mask position with numpy : 0.0067102909088134766 nb_pixel_total : 1404 time to create 1 rle with old method : 0.001680135726928711 time for calcul the mask position with numpy : 0.007340192794799805 nb_pixel_total : 166 time to create 1 rle with old method : 0.0002307891845703125 time for calcul the mask position with numpy : 0.006866455078125 nb_pixel_total : 651 time to create 1 rle with old method : 0.0009241104125976562 time for calcul the mask position with numpy : 0.006920814514160156 nb_pixel_total : 168 time to create 1 rle with old method : 0.00022339820861816406 time for calcul the mask position with numpy : 0.006909370422363281 nb_pixel_total : 1631 time to create 1 rle with old method : 0.002028942108154297 time for calcul the mask position with numpy : 0.008032083511352539 nb_pixel_total : 527 time to create 1 rle with old method : 0.0006573200225830078 time for calcul the mask position with numpy : 0.006925106048583984 nb_pixel_total : 294 time to create 1 rle with old method : 0.0003800392150878906 time for calcul the mask position with numpy : 0.0068514347076416016 nb_pixel_total : 654 time to create 1 rle with old method : 0.0008378028869628906 time for calcul the mask position with numpy : 0.0067365169525146484 nb_pixel_total : 195 time to create 1 rle with old method : 0.0002446174621582031 time for calcul the mask position with numpy : 0.007086515426635742 nb_pixel_total : 729 time to create 1 rle with old method : 0.0008347034454345703 time for calcul the mask position with numpy : 0.008008241653442383 nb_pixel_total : 105937 time to create 1 rle with old method : 0.12265443801879883 time for calcul the mask position with numpy : 0.0068359375 nb_pixel_total : 56 time to create 1 rle with old method : 9.679794311523438e-05 time for calcul the mask position with numpy : 0.006520509719848633 nb_pixel_total : 134 time to create 1 rle with old method : 0.0001823902130126953 time for calcul the mask position with numpy : 0.006601572036743164 nb_pixel_total : 134 time to create 1 rle with old method : 0.00017905235290527344 time for calcul the mask position with numpy : 0.0066258907318115234 nb_pixel_total : 223 time to create 1 rle with old method : 0.0002884864807128906 time for calcul the mask position with numpy : 0.006806135177612305 nb_pixel_total : 203 time to create 1 rle with old method : 0.00029921531677246094 time for calcul the mask position with numpy : 0.0064885616302490234 nb_pixel_total : 313 time to create 1 rle with old method : 0.00045990943908691406 time for calcul the mask position with numpy : 0.006911754608154297 nb_pixel_total : 4465 time to create 1 rle with old method : 0.005329608917236328 time for calcul the mask position with numpy : 0.006770133972167969 nb_pixel_total : 105 time to create 1 rle with old method : 0.00014162063598632812 time for calcul the mask position with numpy : 0.006605386734008789 nb_pixel_total : 405 time to create 1 rle with old method : 0.0004930496215820312 time for calcul the mask position with numpy : 0.0068166255950927734 nb_pixel_total : 261 time to create 1 rle with old method : 0.000339508056640625 time for calcul the mask position with numpy : 0.0064067840576171875 nb_pixel_total : 56 time to create 1 rle with old method : 0.00010895729064941406 time for calcul the mask position with numpy : 0.006506204605102539 nb_pixel_total : 86 time to create 1 rle with old method : 0.00011801719665527344 time for calcul the mask position with numpy : 0.006589412689208984 nb_pixel_total : 355 time to create 1 rle with old method : 0.0004284381866455078 time for calcul the mask position with numpy : 0.006949424743652344 nb_pixel_total : 21 time to create 1 rle with old method : 7.653236389160156e-05 time for calcul the mask position with numpy : 0.008465051651000977 nb_pixel_total : 179 time to create 1 rle with old method : 0.00025582313537597656 time for calcul the mask position with numpy : 0.00777888298034668 nb_pixel_total : 571 time to create 1 rle with old method : 0.0010390281677246094 time for calcul the mask position with numpy : 0.0074939727783203125 nb_pixel_total : 892 time to create 1 rle with old method : 0.001146554946899414 time for calcul the mask position with numpy : 0.006888628005981445 nb_pixel_total : 347 time to create 1 rle with old method : 0.0004291534423828125 time for calcul the mask position with numpy : 0.007158756256103516 nb_pixel_total : 633 time to create 1 rle with old method : 0.0008177757263183594 time for calcul the mask position with numpy : 0.007188558578491211 nb_pixel_total : 114 time to create 1 rle with old method : 0.0002510547637939453 time for calcul the mask position with numpy : 0.007467508316040039 nb_pixel_total : 236 time to create 1 rle with old method : 0.00029349327087402344 time for calcul the mask position with numpy : 0.007356882095336914 nb_pixel_total : 219 time to create 1 rle with old method : 0.00031876564025878906 time for calcul the mask position with numpy : 0.0069730281829833984 nb_pixel_total : 372 time to create 1 rle with old method : 0.00046515464782714844 time for calcul the mask position with numpy : 0.007349491119384766 nb_pixel_total : 289 time to create 1 rle with old method : 0.0004093647003173828 time for calcul the mask position with numpy : 0.006807565689086914 nb_pixel_total : 363 time to create 1 rle with old method : 0.00045371055603027344 time for calcul the mask position with numpy : 0.006596803665161133 nb_pixel_total : 180 time to create 1 rle with old method : 0.00024318695068359375 time for calcul the mask position with numpy : 0.006810903549194336 nb_pixel_total : 327 time to create 1 rle with old method : 0.0005133152008056641 time for calcul the mask position with numpy : 0.007111787796020508 nb_pixel_total : 277 time to create 1 rle with old method : 0.0003514289855957031 time for calcul the mask position with numpy : 0.006567478179931641 nb_pixel_total : 150 time to create 1 rle with old method : 0.00020623207092285156 time for calcul the mask position with numpy : 0.00676727294921875 nb_pixel_total : 194 time to create 1 rle with old method : 0.000263214111328125 time for calcul the mask position with numpy : 0.006577491760253906 nb_pixel_total : 218 time to create 1 rle with old method : 0.00027632713317871094 time for calcul the mask position with numpy : 0.006258487701416016 nb_pixel_total : 543 time to create 1 rle with old method : 0.0006756782531738281 time for calcul the mask position with numpy : 0.006367921829223633 nb_pixel_total : 405 time to create 1 rle with old method : 0.0005197525024414062 time for calcul the mask position with numpy : 0.006438493728637695 nb_pixel_total : 420 time to create 1 rle with old method : 0.0005590915679931641 time for calcul the mask position with numpy : 0.006810426712036133 nb_pixel_total : 15 time to create 1 rle with old method : 3.62396240234375e-05 time for calcul the mask position with numpy : 0.008418083190917969 nb_pixel_total : 103 time to create 1 rle with old method : 0.0001385211944580078 time for calcul the mask position with numpy : 0.008453369140625 nb_pixel_total : 235 time to create 1 rle with old method : 0.0002815723419189453 time for calcul the mask position with numpy : 0.008364439010620117 nb_pixel_total : 970 time to create 1 rle with old method : 0.0012104511260986328 time for calcul the mask position with numpy : 0.008451223373413086 nb_pixel_total : 113 time to create 1 rle with old method : 0.00015473365783691406 time for calcul the mask position with numpy : 0.010345697402954102 nb_pixel_total : 123 time to create 1 rle with old method : 0.0001590251922607422 time for calcul the mask position with numpy : 0.008474349975585938 nb_pixel_total : 83 time to create 1 rle with old method : 0.0001201629638671875 time for calcul the mask position with numpy : 0.008396387100219727 nb_pixel_total : 121 time to create 1 rle with old method : 0.00015592575073242188 create new chi : 2.134552001953125 after preparing all the mask , begin to delete the rle from the crop_hashtag_id => VR 28-11-20 : il faut déplacer cela apres le save_crop_hashtag_ids_obj de la ligne 9514 ! we have 0 chi objets contains the rles time to delete rle : 0.0012671947479248047 batch 1 Loaded 104 chid ids of type : 4230 Number RLEs to save : 10086 TO DO : save crop sub photo not yet done ! save time : 0.6433677673339844 map_output_result : {1339526600: (0.0, 'Should be the crop_list due to order', 0.0), 1339526597: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1339526600, 1339526597] Looping around the photos to save general results len do output : 2 /1339526600.Didn't retrieve data . /1339526597.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2612493') ('4234', '20800049', '1339526600', None, None, None, None, None, '2612493') ('4234', None, None, None, None, None, None, None, '2612493') ('4234', '20800049', '1339526597', None, None, None, None, None, '2612493') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 6 time used for this insertion : 0.014438152313232422 save_final save missing photos in datou_result : time spend for datou_step_exec : 8.978890895843506 time spend to save output : 0.01470041275024414 total time spend for step 4 : 8.99359130859375 step5:crop_condition Tue Feb 25 11:21:08 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 4230 Loading chi in step crop for list_pids : 2 ! batch 1 Loaded 200 chid ids of type : 4230 begin to crop the class : papier param for this class : {'min_score': 0.6} filtre for class : papier hashtag_id of this class : 492668766 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 69 About to insert : list_path_to_insert length 69 new photo from crops ! About to upload 69 photos upload in portfolio : 4869462 init cache_photo without model_param we have 69 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740478869_494175 we have uploaded 69 photos in the portfolio 4869462 time of upload the photos Elapsed time : 24.03738570213318 we have finished the crop for the class : papier begin to crop the class : carton param for this class : {'min_score': 0.6} filtre for class : carton hashtag_id of this class : 492774966 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 30 About to insert : list_path_to_insert length 30 new photo from crops ! About to upload 30 photos upload in portfolio : 4869462 init cache_photo without model_param we have 30 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740478894_494175 we have uploaded 30 photos in the portfolio 4869462 time of upload the photos Elapsed time : 8.33355164527893 we have finished the crop for the class : carton begin to crop the class : metal param for this class : {'min_score': 0.6} filtre for class : metal hashtag_id of this class : 492628673 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740478902_494175 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.604318380355835 we have finished the crop for the class : metal begin to crop the class : pet_clair param for this class : {'min_score': 0.6} filtre for class : pet_clair hashtag_id of this class : 2107755846 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740478903_494175 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.6958489418029785 we have finished the crop for the class : pet_clair begin to crop the class : autre param for this class : {'min_score': 0.6} filtre for class : autre hashtag_id of this class : 494826614 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 22 About to insert : list_path_to_insert length 22 new photo from crops ! About to upload 22 photos upload in portfolio : 4869462 init cache_photo without model_param we have 22 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740478905_494175 we have uploaded 22 photos in the portfolio 4869462 time of upload the photos Elapsed time : 6.35724401473999 we have finished the crop for the class : autre begin to crop the class : pehd param for this class : {'min_score': 0.6} filtre for class : pehd hashtag_id of this class : 628944319 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740478912_494175 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.8536872863769531 we have finished the crop for the class : pehd begin to crop the class : pet_fonce param for this class : {'min_score': 0.6} filtre for class : pet_fonce hashtag_id of this class : 2107755900 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1339526600, 1339526597] Looping around the photos to save general results len do output : 130 /1339528617Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528618Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528619Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528620Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528621Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528622Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528623Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528624Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528625Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528626Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528627Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528628Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528630Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528632Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528633Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528634Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528635Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528636Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528637Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528639Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528640Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528641Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528642Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528643Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528644Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528645Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528647Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528648Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528649Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528650Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528651Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528652Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528653Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528654Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528655Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528656Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528657Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528658Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528659Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528660Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528661Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528662Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528663Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528664Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528665Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528666Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528667Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528668Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528669Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528670Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528671Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528672Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528673Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528674Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528675Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528676Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528677Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528678Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528679Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528680Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528681Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528682Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528683Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528684Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528685Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528686Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528687Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528688Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528689Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528706Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528707Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528708Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528709Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528710Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528711Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528712Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528713Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528714Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528715Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528716Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528717Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528718Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528719Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528720Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528721Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528722Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528723Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528724Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528725Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528726Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528729Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528730Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528731Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528732Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528733Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528734Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528735Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528736Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528737Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528739Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528745Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528746Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528747Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528748Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528749Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528750Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528757Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528758Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528759Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528760Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528761Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528762Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528763Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528764Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528765Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528766Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528767Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528768Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528769Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528771Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528772Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528773Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528774Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528775Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528776Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528777Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528778Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528779Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528783Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1339528784Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2612493') ('4234', '20800049', '1339526600', None, None, None, None, None, '2612493') ('4234', None, None, None, None, None, None, None, '2612493') ('4234', '20800049', '1339526597', None, None, None, None, None, '2612493') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 392 time used for this insertion : 0.03434920310974121 save_final save missing photos in datou_result : time spend for datou_step_exec : 44.643890619277954 time spend to save output : 0.03875994682312012 total time spend for step 5 : 44.682650566101074 step6:thcl Tue Feb 25 11:21:52 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! we are using the classfication for only one thcl 3237 time to import caffe and check if the image exist : 0.007528781890869141 time to convert the images to numpy array : 0.023732423782348633 time to import caffe and check if the image exist : 0.008797168731689453 time to convert the images to numpy array : 0.02974987030029297 time to import caffe and check if the image exist : 0.011722564697265625 time to convert the images to numpy array : 0.02999734878540039 time to import caffe and check if the image exist : 0.011517763137817383 time to convert the images to numpy array : 0.03237199783325195 time to import caffe and check if the image exist : 0.009713888168334961 time to convert the images to numpy array : 0.0336456298828125 time to import caffe and check if the image exist : 0.010061502456665039 time to convert the images to numpy array : 0.03517413139343262 time to import caffe and check if the image exist : 0.013457298278808594 time to convert the images to numpy array : 0.03084540367126465 time to import caffe and check if the image exist : 0.025276899337768555 time to convert the images to numpy array : 0.02079939842224121 time to import caffe and check if the image exist : 0.010647296905517578 time to convert the images to numpy array : 0.03376293182373047 time to import caffe and check if the image exist : 0.017113924026489258 time to convert the images to numpy array : 0.028348922729492188 total time to convert the images to numpy array : 0.37361574172973633 list photo_ids error: [] list photo_ids correct : [1339528673, 1339528674, 1339528675, 1339528676, 1339528677, 1339528678, 1339528679, 1339528680, 1339528681, 1339528682, 1339528683, 1339528684, 1339528685, 1339528660, 1339528661, 1339528662, 1339528663, 1339528664, 1339528665, 1339528666, 1339528667, 1339528668, 1339528669, 1339528670, 1339528671, 1339528672, 1339528686, 1339528687, 1339528688, 1339528689, 1339528706, 1339528707, 1339528708, 1339528709, 1339528710, 1339528711, 1339528712, 1339528713, 1339528714, 1339528617, 1339528618, 1339528619, 1339528620, 1339528621, 1339528622, 1339528623, 1339528624, 1339528625, 1339528626, 1339528627, 1339528628, 1339528630, 1339528730, 1339528731, 1339528732, 1339528733, 1339528734, 1339528735, 1339528736, 1339528737, 1339528739, 1339528745, 1339528746, 1339528747, 1339528748, 1339528647, 1339528648, 1339528649, 1339528650, 1339528651, 1339528652, 1339528653, 1339528654, 1339528655, 1339528656, 1339528657, 1339528658, 1339528659, 1339528715, 1339528716, 1339528717, 1339528718, 1339528719, 1339528720, 1339528721, 1339528722, 1339528723, 1339528724, 1339528725, 1339528726, 1339528729, 1339528632, 1339528633, 1339528634, 1339528635, 1339528636, 1339528637, 1339528639, 1339528640, 1339528641, 1339528642, 1339528643, 1339528644, 1339528645, 1339528768, 1339528769, 1339528771, 1339528772, 1339528773, 1339528774, 1339528775, 1339528776, 1339528777, 1339528778, 1339528779, 1339528783, 1339528784, 1339528749, 1339528750, 1339528757, 1339528758, 1339528759, 1339528760, 1339528761, 1339528762, 1339528763, 1339528764, 1339528765, 1339528766, 1339528767] number of photos to traite : 130 try to delete the photos incorrect in DB tagging for thcl : 3237 To do loadFromThcl(), then load ParamDescType : thcl3237 thcls : [{'id': 3237, 'mtr_user_id': 31, 'name': 'learn_rubbia_refus_2500', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton,Film_plastique,PEHD,PET_clair,PET_fonce,Papier,Tetrapak,flou,mal_croppe,metal,refus', 'svm_portfolios_learning': '4865689,4865690,4865686,4865684,4865685,4865688,4865691,4865693,4865692,4865687,4865683', 'photo_hashtag_type': 4158, 'photo_desc_type': 5561, 'type_classification': 'caffe', 'hashtag_id_list': '492774966,2107756122,628944319,2107755846,2107755900,492668766,609991870,492777938,2107755527,492628673,538914404'}] thcl {'id': 3237, 'mtr_user_id': 31, 'name': 'learn_rubbia_refus_2500', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton,Film_plastique,PEHD,PET_clair,PET_fonce,Papier,Tetrapak,flou,mal_croppe,metal,refus', 'svm_portfolios_learning': '4865689,4865690,4865686,4865684,4865685,4865688,4865691,4865693,4865692,4865687,4865683', 'photo_hashtag_type': 4158, 'photo_desc_type': 5561, 'type_classification': 'caffe', 'hashtag_id_list': '492774966,2107756122,628944319,2107755846,2107755900,492668766,609991870,492777938,2107755527,492628673,538914404'} Update svm_hashtag_type_desc : 5561 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5561, 'learn_rubbia_refus_2500', 2048, 2048, 'learn_rubbia_refus_2500', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2021, 12, 2, 19, 10, 8), datetime.datetime(2021, 12, 2, 19, 10, 8)) To loadFromThcl() : net_5561 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5762 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5561, 'learn_rubbia_refus_2500', 2048, 2048, 'learn_rubbia_refus_2500', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2021, 12, 2, 19, 10, 8), datetime.datetime(2021, 12, 2, 19, 10, 8)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_rubbia_refus_2500 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_rubbia_refus_2500 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_rubbia_refus_2500 /data/models_weight/learn_rubbia_refus_2500/caffemodel size_local : 94358479 size in s3 : 94358479 create time local : 2021-12-03 18:29:39 create time in s3 : 2021-12-02 17:49:16 caffemodel already exist and didn't need to update /data/models_weight/learn_rubbia_refus_2500/deploy.prototxt size_local : 32544 size in s3 : 32544 create time local : 2021-12-03 18:29:39 create time in s3 : 2021-12-02 17:49:15 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_rubbia_refus_2500/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-12-03 18:29:40 create time in s3 : 2021-12-02 18:09:52 mean.npy already exist and didn't need to update /data/models_weight/learn_rubbia_refus_2500/synset_words.txt size_local : 334 size in s3 : 334 create time local : 2021-12-03 18:29:40 create time in s3 : 2021-12-02 18:10:06 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_rubbia_refus_2500/deploy.prototxt caffemodel_filename : /data/models_weight/learn_rubbia_refus_2500/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5543 max_wait_temp : 1 max_wait : 0 dict_keys(['pool5', 'prob']) time used to do the prepocess of the images : 0.6551742553710938 time used to do the prediction : 0.4863593578338623 save descriptor for thcl : 3237 time to traite the descriptors : 0.7969074249267578 Catched exception ! Connect or reconnect ! storage_type for insertDescriptorsMulti : 3 To insert : 1339528673 To insert : 1339528674 To insert : 1339528675 To insert : 1339528676 To insert : 1339528677 To insert : 1339528678 To insert : 1339528679 To insert : 1339528680 To insert : 1339528681 To insert : 1339528682 To insert : 1339528683 To insert : 1339528684 To insert : 1339528685 To insert : 1339528660 To insert : 1339528661 To insert : 1339528662 To insert : 1339528663 To insert : 1339528664 To insert : 1339528665 To insert : 1339528666 To insert : 1339528667 To insert : 1339528668 To insert : 1339528669 To insert : 1339528670 To insert : 1339528671 To insert : 1339528672 To insert : 1339528686 To insert : 1339528687 To insert : 1339528688 To insert : 1339528689 To insert : 1339528706 To insert : 1339528707 To insert : 1339528708 To insert : 1339528709 To insert : 1339528710 To insert : 1339528711 To insert : 1339528712 To insert : 1339528713 To insert : 1339528714 To insert : 1339528617 To insert : 1339528618 To insert : 1339528619 To insert : 1339528620 To insert : 1339528621 To insert : 1339528622 To insert : 1339528623 To insert : 1339528624 To insert : 1339528625 To insert : 1339528626 To insert : 1339528627 To insert : 1339528628 To insert : 1339528630 To insert : 1339528730 To insert : 1339528731 To insert : 1339528732 To insert : 1339528733 To insert : 1339528734 To insert : 1339528735 To insert : 1339528736 To insert : 1339528737 To insert : 1339528739 To insert : 1339528745 To insert : 1339528746 To insert : 1339528747 To insert : 1339528748 To insert : 1339528647 To insert : 1339528648 To insert : 1339528649 To insert : 1339528650 To insert : 1339528651 To insert : 1339528652 To insert : 1339528653 To insert : 1339528654 To insert : 1339528655 To insert : 1339528656 To insert : 1339528657 To insert : 1339528658 To insert : 1339528659 To insert : 1339528715 To insert : 1339528716 To insert : 1339528717 To insert : 1339528718 To insert : 1339528719 To insert : 1339528720 To insert : 1339528721 To insert : 1339528722 To insert : 1339528723 To insert : 1339528724 To insert : 1339528725 To insert : 1339528726 To insert : 1339528729 To insert : 1339528632 To insert : 1339528633 To insert : 1339528634 To insert : 1339528635 To insert : 1339528636 To insert : 1339528637 To insert : 1339528639 To insert : 1339528640 To insert : 1339528641 To insert : 1339528642 To insert : 1339528643 To insert : 1339528644 To insert : 1339528645 To insert : 1339528768 To insert : 1339528769 To insert : 1339528771 To insert : 1339528772 To insert : 1339528773 To insert : 1339528774 To insert : 1339528775 To insert : 1339528776 To insert : 1339528777 To insert : 1339528778 To insert : 1339528779 To insert : 1339528783 To insert : 1339528784 To insert : 1339528749 To insert : 1339528750 To insert : 1339528757 To insert : 1339528758 To insert : 1339528759 To insert : 1339528760 To insert : 1339528761 To insert : 1339528762 To insert : 1339528763 To insert : 1339528764 To insert : 1339528765 To insert : 1339528766 To insert : 1339528767 time to insert the descriptors : 26.017082929611206 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos SAVE THCL : begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 199 time used for this insertion : 0.020092487335205078 save missing photos in datou_result : time spend for datou_step_exec : 32.27091431617737 time spend to save output : 0.0653831958770752 total time spend for step 6 : 32.33629751205444 step7:ventilate_hashtags_in_portfolio Tue Feb 25 11:22:25 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! Iterating over portfolio : 20800049 get user id for portfolio 20800049 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20800049 AND mptpi.`type`=4230 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','mal_croppe','PET_fonce','PEHD','refus','Carton','autre','Film_plastique','carton','metal','environnement','Tetrapak','pet_fonce','Papier','pehd','PET_clair','pet_clair','flou')) AND mptpi.`min_score`=0.6 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20800049 AND mptpi.`type`=4230 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','mal_croppe','PET_fonce','PEHD','refus','Carton','autre','Film_plastique','carton','metal','environnement','Tetrapak','pet_fonce','Papier','pehd','PET_clair','pet_clair','flou')) AND mptpi.`min_score`=0.6 To do To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20800049 AND mptpi.`type`=4231 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','mal_croppe','PET_fonce','PEHD','refus','Carton','autre','Film_plastique','carton','metal','environnement','Tetrapak','pet_fonce','Papier','pehd','PET_clair','pet_clair','flou')) AND mptpi.`min_score`=0.6 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20800049 AND mptpi.`type`=4231 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','mal_croppe','PET_fonce','PEHD','refus','Carton','autre','Film_plastique','carton','metal','environnement','Tetrapak','pet_fonce','Papier','pehd','PET_clair','pet_clair','flou')) AND mptpi.`min_score`=0.6 To do lien utilise dans velours : https://www.fotonower.com/velours/20800677,20800665,20800676,20800678,20800668,20800672,20800670,20800671,20800673,20800674,20800675,20800680,20800681?tags=papier,mal_croppe,pet_fonce,pehd,refus,carton,autre,film_plastique,metal,environnement,tetrapak,pet_clair,flou&datou_id_consolidate=4235&port_consolidate=20800049 Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1339526600, 1339526597] Looping around the photos to save general results len do output : 1 /20800049. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2612493') ('4234', '20800049', '1339526600', None, None, None, None, None, '2612493') ('4234', None, None, None, None, None, None, None, '2612493') ('4234', '20800049', '1339526597', None, None, None, None, None, '2612493') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 time used for this insertion : 0.015714406967163086 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.5967013835906982 time spend to save output : 0.015992403030395508 total time spend for step 7 : 1.6126937866210938 step8:final Tue Feb 25 11:22:26 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1339526600: ('0.1438923595798674',), 1339526597: ('0.1438923595798674',)} new output for save of step final : {1339526600: ('0.1438923595798674',), 1339526597: ('0.1438923595798674',)} [1339526600, 1339526597] Looping around the photos to save general results len do output : 2 /1339526600.Didn't retrieve data . /1339526597.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2612493') ('4234', '20800049', '1339526600', None, None, None, None, None, '2612493') ('4234', None, None, None, None, None, None, None, '2612493') ('4234', '20800049', '1339526597', None, None, None, None, None, '2612493') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 6 time used for this insertion : 0.015241146087646484 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.11705636978149414 time spend to save output : 0.01549673080444336 total time spend for step 8 : 0.1325531005859375 step9:velours_tree Tue Feb 25 11:22:26 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure list_portfolios : 20800677,20800665,20800676,20800678,20800668,20800672,20800670,20800671,20800673,20800674,20800675,20800680,20800681 photo desc type : 5561 - Retrieving photos to tag... query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20800677 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20800665 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20800676 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20800678 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20800668 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20800672 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20800670 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20800671 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20800673 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20800674 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20800675 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20800680 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20800681 ORDER BY ph.size desc - Loading descriptors... Size : 2048 len(descriptors) : 130 Compute structured hierarchical clustering... ward : AgglomerativeClustering(n_clusters=6) ward.labels_ : [3 0 2 2 3 2 0 0 2 1 0 0 1 5 0 0 0 0 0 1 1 0 0 0 0 0 0 5 5 1 5 1 0 5 2 0 3 0 2 1 0 0 0 0 1 2 0 0 1 0 2 0 0 0 0 0 1 1 0 0 0 0 0 5 0 0 5 1 2 3 3 0 0 1 4 0 4 0 0 2 2 2 2 5 3 3 2 1 1 0 0 2 0 2 0 2 2 1 0 2 2 2 2 2 2 2 0 0 5 5 0 0 5 5 1 2 4 0 0 0 5 5 5 0 5 5 5 5 0 0] Elapsed time: 0.018162012100219727 graph_id used : 80208 - Beta version, working pretty good on 11-5-16 ! https://www.fotonower.com/velours/20800677,20800665,20800676,20800678,20800668,20800672,20800670,20800671,20800673,20800674,20800675,20800680,20800681?tags=papier,mal_croppe,pet_fonce,pehd,refus,carton,autre,film_plastique,metal,environnement,tetrapak,pet_clair,flou&datou_id_consolidate=4235&port_consolidate=20800049&tree_id=80208 Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : velours_tree we use saveGeneral [1339526600, 1339526597] Looping around the photos to save general results len do output : 1 /20800049Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2612493') ('4234', '20800049', '1339526600', None, None, None, None, None, '2612493') ('4234', None, None, None, None, None, None, None, '2612493') ('4234', '20800049', '1339526597', None, None, None, None, None, '2612493') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 time used for this insertion : 0.4406721591949463 save_final save missing photos in datou_result : time spend for datou_step_exec : 11.470140218734741 time spend to save output : 0.44105029106140137 total time spend for step 9 : 11.911190509796143 step10:send_mail_cod Tue Feb 25 11:22:38 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Qualipapia_P20800049_25-02-2025_11_22_38.pdf 20800659 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette208006591740478958 20800647 imagette208006471740478959 20800658 imagette208006581740478959 20800660 change filename to text .change filename to text .imagette208006601740478959 20800650 imagette208006501740478959 20800654 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette208006541740478959 20800652 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette208006521740478961 20800653 imagette208006531740478962 20800655 change filename to text .imagette208006551740478962 20800657 imagette208006571740478962 20800662 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette208006621740478962 20800663 imagette208006631740478962 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20800049 and hashtag_type = 4230 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20800677,20800665,20800676,20800678,20800668,20800672,20800670,20800671,20800673,20800674,20800675,20800680,20800681?tags=papier,mal_croppe,pet_fonce,pehd,refus,carton,autre,film_plastique,metal,environnement,tetrapak,pet_clair,flou&datou_id_consolidate=4235&port_consolidate=20800049&tree_id=80208 your option no_mail is active, we will not send the real mail to your client args[1339526600] : ((1339526600, -7.134587080315985, 492609224), (1339526600, -0.3634676620777829, 496442774), '0.1438923595798674') We are sending mail with results at cod@fotonower.com args[1339526597] : ((1339526597, -7.0980378304370095, 492609224), (1339526597, -0.3583013155559034, 496442774), '0.1438923595798674') We are sending mail with results at cod@fotonower.com refus_total : 0.1438923595798674 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20800049 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20800049_25-02-2025_11_22_38.pdf results_Qualipapia_P20800049_25-02-2025_11_22_38.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20800049_25-02-2025_11_22_38.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('4234','20800049','results_Qualipapia_P20800049_25-02-2025_11_22_38.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20800049_25-02-2025_11_22_38.pdf','pdf','','0.14','0.1438923595798674') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1339526600, 1339526597] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2612493') ('4234', '20800049', '1339526600', None, None, None, None, None, '2612493') ('4234', None, None, None, None, None, None, None, '2612493') ('4234', '20800049', '1339526597', None, None, None, None, None, '2612493') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 2 time used for this insertion : 0.015879392623901367 save_final save missing photos in datou_result : time spend for datou_step_exec : 5.094480752944946 time spend to save output : 0.016066312789916992 total time spend for step 10 : 5.110547065734863 step11:split_time_score Tue Feb 25 11:22:43 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 3442, 'mtr_user_id': 31, 'name': 'classifieur_2camions_valcor_021122_v1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'deux_camions,camion_droite,camion_gauche,pas_de_camion', 'svm_portfolios_learning': '7659379,7659034,7657685,7657114', 'photo_hashtag_type': 4458, 'photo_desc_type': 5723, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107760533,2107760534,2107760535,2107760536'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('08', 2),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 25022025 20800049 Nombre de photos uploadées : 2 / 23040 (0%) 25022025 20800049 Nombre de photos taguées (types de déchets): 2 / 2 (100%) 25022025 20800049 Nombre de photos taguées (volume) : 0 / 2 (0%) elapsed_time : load_data_split_time_score 3.0994415283203125e-06 elapsed_time : order_list_meta_photo_and_scores 1.8358230590820312e-05 ?? elapsed_time : fill_and_build_computed_from_old_data 0.00039577484130859375 elapsed_time : insert_dashboard_record_day_entry 0.025019168853759766 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20800048 order by id desc limit 1 Qualite : 0.1438923595798674 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20800049_25-02-2025_11_22_38.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20800049 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20800049 AND mptpi.`type`=4230 To do Qualite : 0.13019775739708864 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20800050_25-02-2025_11_18_27.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20800050 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20800050 AND mptpi.`type`=4230 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'25022025': {'nb_upload': 2, 'nb_taggue_class': 2, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1339526600, 1339526597] Looping around the photos to save general results len do output : 1 /20800049Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2612493') ('4234', '20800049', '1339526600', None, None, None, None, None, '2612493') ('4234', None, None, None, None, None, None, None, '2612493') ('4234', '20800049', '1339526597', None, None, None, None, None, '2612493') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 time used for this insertion : 0.139617919921875 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.0883746147155762 time spend to save output : 0.13982343673706055 total time spend for step 11 : 1.2281980514526367 caffe_path_current : About to save ! 2 After save, about to update current ! ret : 1 len(input) + len(total_photo_id_missing) : 2 set_done_treatment 34.39user 23.93system 2:20.44elapsed 41%CPU (0avgtext+0avgdata 2465684maxresident)k 748936inputs+10400outputs (515major+2145795minor)pagefaults 0swaps