python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 4234 -l 5 ' -s datou_current_4234 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 4169668 load datou : 4234 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5275, 'learn_RUBBIA_REFUS_AMIENS_23', 16384, 25088, 'learn_RUBBIA_REFUS_AMIENS_23', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 23, 14, 19, 39), datetime.datetime(2021, 4, 23, 14, 19, 39)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5561, 'learn_rubbia_refus_2500', 2048, 2048, 'learn_rubbia_refus_2500', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2021, 12, 2, 19, 10, 8), datetime.datetime(2021, 12, 2, 19, 10, 8)) load thcls load THCL from format json or kwargs add thcl : 2847 in CacheModelConfig load THCL from format json or kwargs add thcl : 3237 in CacheModelConfig load pdts add pdt : 5275 in CacheModelConfig add pdt : 5561 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 list_input_json: [] Current got : datou_id : 4234, datou_cur_ids : ['2545063'] with mtr_portfolio_ids : ['20210830'] and first list_photo_ids : [] new path : /proc/4169668/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, brightness, blur_detection, rle_unique_nms_with_priority, crop_condition, thcl, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , BFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 2 ; length of list_pids : 2 ; length of list_args : 2 time to download the photos : 0.25757312774658203 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Tue Feb 4 14:20:28 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10553 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-04 14:20:32.563612: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-04 14:20:32.595117: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-04 14:20:32.597295: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f6318000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-04 14:20:32.597344: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-04 14:20:32.601682: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-04 14:20:32.859631: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1ab3e190 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-04 14:20:32.859689: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-04 14:20:32.861139: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-04 14:20:32.861919: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-04 14:20:32.867014: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-04 14:20:32.870061: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-04 14:20:32.871710: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-04 14:20:32.875997: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-04 14:20:32.877629: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-04 14:20:32.884480: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-04 14:20:32.885932: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-04 14:20:32.885995: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-04 14:20:32.886755: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-04 14:20:32.886770: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-04 14:20:32.886779: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-04 14:20:32.888169: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9777 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-04 14:20:33.185601: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-04 14:20:33.185710: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-04 14:20:33.185742: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-04 14:20:33.185771: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-04 14:20:33.185799: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-04 14:20:33.185853: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-04 14:20:33.185883: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-04 14:20:33.185911: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-04 14:20:33.188066: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-04 14:20:33.189379: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-04 14:20:33.189420: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-04 14:20:33.189438: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-04 14:20:33.189454: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-04 14:20:33.189470: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-04 14:20:33.189486: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-04 14:20:33.189501: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-04 14:20:33.189517: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-04 14:20:33.190879: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-04 14:20:33.190924: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-04 14:20:33.190947: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-04 14:20:33.190955: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-04 14:20:33.192371: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9777 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2847 thcls : [{'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}] thcl {'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5275 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5275, 'learn_RUBBIA_REFUS_AMIENS_23', 16384, 25088, 'learn_RUBBIA_REFUS_AMIENS_23', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 23, 14, 19, 39), datetime.datetime(2021, 4, 23, 14, 19, 39)) {'thcl': {'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'], 'list_hashtags_csv': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'svm_hashtag_type_desc': 5275, 'photo_desc_type': 5275, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_RUBBIA_REFUS_AMIENS_23 NUM_CLASSES 9 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_RUBBIA_REFUS_AMIENS_23 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-04 14:20:40.392683: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-04 14:20:40.575714: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_RUBBIA_REFUS_AMIENS_23 /data/models_weight/learn_RUBBIA_REFUS_AMIENS_23/mask_model.h5 size_local : 256009536 size in s3 : 256009536 create time local : 2021-08-09 09:43:22 create time in s3 : 2021-08-06 18:54:04 mask_model.h5 already exist and didn't need to update list_images length : 2 NEW PHOTO pour l'instant on ne peut pas sauvegarder la photo dans les tile Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -118.36406 max: 149.57656 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 8 time for calcul the mask position with numpy : 6.961822509765625e-05 nb_pixel_total : 429 time to create 1 rle with old method : 0.000820159912109375 length of segment : 27 time for calcul the mask position with numpy : 5.7220458984375e-05 nb_pixel_total : 812 time to create 1 rle with old method : 0.0014755725860595703 length of segment : 29 time for calcul the mask position with numpy : 4.172325134277344e-05 nb_pixel_total : 83 time to create 1 rle with old method : 0.0001804828643798828 length of segment : 28 time for calcul the mask position with numpy : 5.125999450683594e-05 nb_pixel_total : 717 time to create 1 rle with old method : 0.0013628005981445312 length of segment : 37 time for calcul the mask position with numpy : 7.05718994140625e-05 nb_pixel_total : 1738 time to create 1 rle with old method : 0.003032207489013672 length of segment : 49 time for calcul the mask position with numpy : 4.124641418457031e-05 nb_pixel_total : 98 time to create 1 rle with old method : 0.00020170211791992188 length of segment : 29 time for calcul the mask position with numpy : 5.7220458984375e-05 nb_pixel_total : 898 time to create 1 rle with old method : 0.0016505718231201172 length of segment : 37 time for calcul the mask position with numpy : 6.961822509765625e-05 nb_pixel_total : 2207 time to create 1 rle with old method : 0.003845691680908203 length of segment : 45 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -115.08672 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 12 time for calcul the mask position with numpy : 4.506111145019531e-05 nb_pixel_total : 410 time to create 1 rle with old method : 0.0005409717559814453 length of segment : 29 time for calcul the mask position with numpy : 3.6716461181640625e-05 nb_pixel_total : 387 time to create 1 rle with old method : 0.0005145072937011719 length of segment : 37 time for calcul the mask position with numpy : 6.508827209472656e-05 nb_pixel_total : 2614 time to create 1 rle with old method : 0.003292560577392578 length of segment : 50 time for calcul the mask position with numpy : 3.266334533691406e-05 nb_pixel_total : 56 time to create 1 rle with old method : 0.00011157989501953125 length of segment : 16 time for calcul the mask position with numpy : 0.0001323223114013672 nb_pixel_total : 7197 time to create 1 rle with old method : 0.008580684661865234 length of segment : 69 time for calcul the mask position with numpy : 4.6253204345703125e-05 nb_pixel_total : 762 time to create 1 rle with old method : 0.0009748935699462891 length of segment : 38 time for calcul the mask position with numpy : 4.076957702636719e-05 nb_pixel_total : 855 time to create 1 rle with old method : 0.0011420249938964844 length of segment : 30 time for calcul the mask position with numpy : 4.315376281738281e-05 nb_pixel_total : 916 time to create 1 rle with old method : 0.0011773109436035156 length of segment : 40 time for calcul the mask position with numpy : 3.314018249511719e-05 nb_pixel_total : 82 time to create 1 rle with old method : 0.0001246929168701172 length of segment : 15 time for calcul the mask position with numpy : 4.0531158447265625e-05 nb_pixel_total : 710 time to create 1 rle with old method : 0.000949859619140625 length of segment : 40 time for calcul the mask position with numpy : 6.4849853515625e-05 nb_pixel_total : 1890 time to create 1 rle with old method : 0.0022275447845458984 length of segment : 85 time for calcul the mask position with numpy : 0.000152587890625 nb_pixel_total : 8218 time to create 1 rle with old method : 0.009807109832763672 length of segment : 74 Processing 1 images image shape: (400, 400, 3) min: 4.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -97.37969 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 3 time for calcul the mask position with numpy : 0.00011229515075683594 nb_pixel_total : 2655 time to create 1 rle with old method : 0.0034537315368652344 length of segment : 118 time for calcul the mask position with numpy : 0.000118255615234375 nb_pixel_total : 2368 time to create 1 rle with old method : 0.003249645233154297 length of segment : 117 time for calcul the mask position with numpy : 0.00010418891906738281 nb_pixel_total : 2592 time to create 1 rle with old method : 0.0035600662231445312 length of segment : 31 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -120.75469 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 7 time for calcul the mask position with numpy : 4.553794860839844e-05 nb_pixel_total : 93 time to create 1 rle with old method : 0.00022721290588378906 length of segment : 10 time for calcul the mask position with numpy : 5.9604644775390625e-05 nb_pixel_total : 1186 time to create 1 rle with old method : 0.006780385971069336 length of segment : 62 time for calcul the mask position with numpy : 0.00015544891357421875 nb_pixel_total : 5623 time to create 1 rle with old method : 0.01761651039123535 length of segment : 119 time for calcul the mask position with numpy : 3.6716461181640625e-05 nb_pixel_total : 181 time to create 1 rle with old method : 0.0002605915069580078 length of segment : 18 time for calcul the mask position with numpy : 3.0994415283203125e-05 nb_pixel_total : 172 time to create 1 rle with old method : 0.00028395652770996094 length of segment : 23 time for calcul the mask position with numpy : 4.172325134277344e-05 nb_pixel_total : 182 time to create 1 rle with old method : 0.0003528594970703125 length of segment : 24 time for calcul the mask position with numpy : 4.267692565917969e-05 nb_pixel_total : 267 time to create 1 rle with old method : 0.0004992485046386719 length of segment : 30 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 123.86250 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 6 time for calcul the mask position with numpy : 9.012222290039062e-05 nb_pixel_total : 1774 time to create 1 rle with old method : 0.002552509307861328 length of segment : 128 time for calcul the mask position with numpy : 4.363059997558594e-05 nb_pixel_total : 225 time to create 1 rle with old method : 0.0004398822784423828 length of segment : 19 time for calcul the mask position with numpy : 4.506111145019531e-05 nb_pixel_total : 1023 time to create 1 rle with old method : 0.001344442367553711 length of segment : 42 time for calcul the mask position with numpy : 3.647804260253906e-05 nb_pixel_total : 356 time to create 1 rle with old method : 0.0004885196685791016 length of segment : 22 time for calcul the mask position with numpy : 4.124641418457031e-05 nb_pixel_total : 372 time to create 1 rle with old method : 0.0006363391876220703 length of segment : 21 time for calcul the mask position with numpy : 0.0006091594696044922 nb_pixel_total : 48664 time to create 1 rle with old method : 0.05724906921386719 length of segment : 212 Processing 1 images image shape: (400, 400, 3) min: 2.00000 max: 212.00000 molded_images shape: (1, 640, 640, 3) min: -94.27109 max: 69.99141 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 0 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -117.11797 max: 135.47344 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 6 time for calcul the mask position with numpy : 4.076957702636719e-05 nb_pixel_total : 437 time to create 1 rle with old method : 0.000629425048828125 length of segment : 24 time for calcul the mask position with numpy : 0.0003662109375 nb_pixel_total : 30553 time to create 1 rle with old method : 0.03582143783569336 length of segment : 254 time for calcul the mask position with numpy : 5.1975250244140625e-05 nb_pixel_total : 308 time to create 1 rle with old method : 0.0006072521209716797 length of segment : 21 time for calcul the mask position with numpy : 5.14984130859375e-05 nb_pixel_total : 649 time to create 1 rle with old method : 0.001222848892211914 length of segment : 29 time for calcul the mask position with numpy : 4.57763671875e-05 nb_pixel_total : 249 time to create 1 rle with old method : 0.000469207763671875 length of segment : 26 time for calcul the mask position with numpy : 7.486343383789062e-05 nb_pixel_total : 1421 time to create 1 rle with old method : 0.0025000572204589844 length of segment : 61 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -106.14922 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 11 time for calcul the mask position with numpy : 4.363059997558594e-05 nb_pixel_total : 152 time to create 1 rle with old method : 0.0002598762512207031 length of segment : 24 time for calcul the mask position with numpy : 5.7697296142578125e-05 nb_pixel_total : 2186 time to create 1 rle with old method : 0.002728700637817383 length of segment : 40 time for calcul the mask position with numpy : 5.745887756347656e-05 nb_pixel_total : 1895 time to create 1 rle with old method : 0.002268552780151367 length of segment : 49 time for calcul the mask position with numpy : 9.012222290039062e-05 nb_pixel_total : 3329 time to create 1 rle with old method : 0.004099369049072266 length of segment : 90 time for calcul the mask position with numpy : 3.504753112792969e-05 nb_pixel_total : 133 time to create 1 rle with old method : 0.00022292137145996094 length of segment : 14 time for calcul the mask position with numpy : 0.00017499923706054688 nb_pixel_total : 10832 time to create 1 rle with old method : 0.012756824493408203 length of segment : 213 time for calcul the mask position with numpy : 3.9577484130859375e-05 nb_pixel_total : 366 time to create 1 rle with old method : 0.0004928112030029297 length of segment : 20 time for calcul the mask position with numpy : 3.719329833984375e-05 nb_pixel_total : 616 time to create 1 rle with old method : 0.0007658004760742188 length of segment : 29 time for calcul the mask position with numpy : 5.221366882324219e-05 nb_pixel_total : 1052 time to create 1 rle with old method : 0.001405477523803711 length of segment : 50 time for calcul the mask position with numpy : 3.123283386230469e-05 nb_pixel_total : 61 time to create 1 rle with old method : 0.00012135505676269531 length of segment : 8 time for calcul the mask position with numpy : 5.841255187988281e-05 nb_pixel_total : 2342 time to create 1 rle with old method : 0.0029573440551757812 length of segment : 43 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -118.23516 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 8 time for calcul the mask position with numpy : 0.00010824203491210938 nb_pixel_total : 3656 time to create 1 rle with old method : 0.006113767623901367 length of segment : 134 time for calcul the mask position with numpy : 5.4836273193359375e-05 nb_pixel_total : 453 time to create 1 rle with old method : 0.0008385181427001953 length of segment : 45 time for calcul the mask position with numpy : 4.839897155761719e-05 nb_pixel_total : 431 time to create 1 rle with old method : 0.0008733272552490234 length of segment : 26 time for calcul the mask position with numpy : 0.00026607513427734375 nb_pixel_total : 4494 time to create 1 rle with old method : 0.008626222610473633 length of segment : 254 time for calcul the mask position with numpy : 4.029273986816406e-05 nb_pixel_total : 256 time to create 1 rle with old method : 0.0004239082336425781 length of segment : 25 time for calcul the mask position with numpy : 3.24249267578125e-05 nb_pixel_total : 124 time to create 1 rle with old method : 0.00018835067749023438 length of segment : 25 time for calcul the mask position with numpy : 8.153915405273438e-05 nb_pixel_total : 3812 time to create 1 rle with old method : 0.004470348358154297 length of segment : 100 time for calcul the mask position with numpy : 0.00019550323486328125 nb_pixel_total : 3264 time to create 1 rle with old method : 0.004286766052246094 length of segment : 107 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 134.86016 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 10 time for calcul the mask position with numpy : 4.506111145019531e-05 nb_pixel_total : 389 time to create 1 rle with old method : 0.0004971027374267578 length of segment : 31 time for calcul the mask position with numpy : 6.866455078125e-05 nb_pixel_total : 448 time to create 1 rle with old method : 0.0006377696990966797 length of segment : 72 time for calcul the mask position with numpy : 6.699562072753906e-05 nb_pixel_total : 1867 time to create 1 rle with old method : 0.002307891845703125 length of segment : 89 time for calcul the mask position with numpy : 6.914138793945312e-05 nb_pixel_total : 1318 time to create 1 rle with old method : 0.0017268657684326172 length of segment : 118 time for calcul the mask position with numpy : 9.250640869140625e-05 nb_pixel_total : 2140 time to create 1 rle with old method : 0.0026330947875976562 length of segment : 118 time for calcul the mask position with numpy : 3.361701965332031e-05 nb_pixel_total : 101 time to create 1 rle with old method : 0.00016427040100097656 length of segment : 19 time for calcul the mask position with numpy : 0.00010609626770019531 nb_pixel_total : 1795 time to create 1 rle with old method : 0.0022225379943847656 length of segment : 91 time for calcul the mask position with numpy : 3.4809112548828125e-05 nb_pixel_total : 286 time to create 1 rle with old method : 0.0003902912139892578 length of segment : 28 time for calcul the mask position with numpy : 3.314018249511719e-05 nb_pixel_total : 308 time to create 1 rle with old method : 0.0004489421844482422 length of segment : 30 time for calcul the mask position with numpy : 6.365776062011719e-05 nb_pixel_total : 1785 time to create 1 rle with old method : 0.0021581649780273438 length of segment : 90 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 187.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 61.46172 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 1 time for calcul the mask position with numpy : 0.0008728504180908203 nb_pixel_total : 106476 time to create 1 rle with old method : 0.12407946586608887 length of segment : 279 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 148.61562 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 10 time for calcul the mask position with numpy : 8.034706115722656e-05 nb_pixel_total : 1490 time to create 1 rle with old method : 0.0025925636291503906 length of segment : 45 time for calcul the mask position with numpy : 0.0008072853088378906 nb_pixel_total : 52089 time to create 1 rle with old method : 0.07302689552307129 length of segment : 439 time for calcul the mask position with numpy : 8.797645568847656e-05 nb_pixel_total : 570 time to create 1 rle with old method : 0.0008380413055419922 length of segment : 45 time for calcul the mask position with numpy : 0.0001430511474609375 nb_pixel_total : 4881 time to create 1 rle with old method : 0.005971431732177734 length of segment : 85 time for calcul the mask position with numpy : 5.435943603515625e-05 nb_pixel_total : 1068 time to create 1 rle with old method : 0.0013124942779541016 length of segment : 63 length of segment : 0 time for calcul the mask position with numpy : 4.1484832763671875e-05 nb_pixel_total : 429 time to create 1 rle with old method : 0.0005829334259033203 length of segment : 30 time for calcul the mask position with numpy : 3.0994415283203125e-05 nb_pixel_total : 115 time to create 1 rle with old method : 0.00018739700317382812 length of segment : 11 time for calcul the mask position with numpy : 6.413459777832031e-05 nb_pixel_total : 352 time to create 1 rle with old method : 0.0005609989166259766 length of segment : 73 time for calcul the mask position with numpy : 5.53131103515625e-05 nb_pixel_total : 150 time to create 1 rle with old method : 0.0002582073211669922 length of segment : 20 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 143.13125 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 8 time for calcul the mask position with numpy : 5.125999450683594e-05 nb_pixel_total : 1006 time to create 1 rle with old method : 0.0013535022735595703 length of segment : 34 time for calcul the mask position with numpy : 4.9591064453125e-05 nb_pixel_total : 1183 time to create 1 rle with old method : 0.0014481544494628906 length of segment : 62 time for calcul the mask position with numpy : 6.890296936035156e-05 nb_pixel_total : 3321 time to create 1 rle with old method : 0.003801584243774414 length of segment : 65 time for calcul the mask position with numpy : 4.291534423828125e-05 nb_pixel_total : 778 time to create 1 rle with old method : 0.0009686946868896484 length of segment : 35 time for calcul the mask position with numpy : 4.076957702636719e-05 nb_pixel_total : 497 time to create 1 rle with old method : 0.0007479190826416016 length of segment : 37 time for calcul the mask position with numpy : 3.24249267578125e-05 nb_pixel_total : 174 time to create 1 rle with old method : 0.00025582313537597656 length of segment : 12 time for calcul the mask position with numpy : 5.9604644775390625e-05 nb_pixel_total : 1974 time to create 1 rle with old method : 0.0024001598358154297 length of segment : 53 time for calcul the mask position with numpy : 3.838539123535156e-05 nb_pixel_total : 351 time to create 1 rle with old method : 0.00046706199645996094 length of segment : 26 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 149.92812 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 11 time for calcul the mask position with numpy : 4.124641418457031e-05 nb_pixel_total : 332 time to create 1 rle with old method : 0.0004773139953613281 length of segment : 20 time for calcul the mask position with numpy : 3.838539123535156e-05 nb_pixel_total : 458 time to create 1 rle with old method : 0.0006868839263916016 length of segment : 29 time for calcul the mask position with numpy : 6.270408630371094e-05 nb_pixel_total : 984 time to create 1 rle with old method : 0.0012214183807373047 length of segment : 61 time for calcul the mask position with numpy : 4.696846008300781e-05 nb_pixel_total : 47 time to create 1 rle with old method : 0.000102996826171875 length of segment : 9 time for calcul the mask position with numpy : 6.67572021484375e-05 nb_pixel_total : 3175 time to create 1 rle with old method : 0.003955841064453125 length of segment : 84 time for calcul the mask position with numpy : 5.054473876953125e-05 nb_pixel_total : 1031 time to create 1 rle with old method : 0.001245737075805664 length of segment : 62 time for calcul the mask position with numpy : 3.2901763916015625e-05 nb_pixel_total : 244 time to create 1 rle with old method : 0.0003619194030761719 length of segment : 14 time for calcul the mask position with numpy : 3.409385681152344e-05 nb_pixel_total : 294 time to create 1 rle with old method : 0.0003921985626220703 length of segment : 29 time for calcul the mask position with numpy : 3.9577484130859375e-05 nb_pixel_total : 203 time to create 1 rle with old method : 0.0003476142883300781 length of segment : 28 time for calcul the mask position with numpy : 8.487701416015625e-05 nb_pixel_total : 3587 time to create 1 rle with old method : 0.004370212554931641 length of segment : 93 time for calcul the mask position with numpy : 4.601478576660156e-05 nb_pixel_total : 915 time to create 1 rle with old method : 0.0011031627655029297 length of segment : 56 Processing 1 images image shape: (280, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 134.95000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 1 time for calcul the mask position with numpy : 7.748603820800781e-05 nb_pixel_total : 101 time to create 1 rle with old method : 0.0002734661102294922 length of segment : 39 NEW PHOTO pour l'instant on ne peut pas sauvegarder la photo dans les tile Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -118.91484 max: 150.15078 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 7 time for calcul the mask position with numpy : 5.316734313964844e-05 nb_pixel_total : 411 time to create 1 rle with old method : 0.0007781982421875 length of segment : 28 time for calcul the mask position with numpy : 5.435943603515625e-05 nb_pixel_total : 736 time to create 1 rle with old method : 0.0014717578887939453 length of segment : 33 time for calcul the mask position with numpy : 4.887580871582031e-05 nb_pixel_total : 732 time to create 1 rle with old method : 0.0014042854309082031 length of segment : 28 time for calcul the mask position with numpy : 3.9577484130859375e-05 nb_pixel_total : 95 time to create 1 rle with old method : 0.00021338462829589844 length of segment : 27 time for calcul the mask position with numpy : 5.888938903808594e-05 nb_pixel_total : 1221 time to create 1 rle with old method : 0.0023529529571533203 length of segment : 60 time for calcul the mask position with numpy : 4.2438507080078125e-05 nb_pixel_total : 89 time to create 1 rle with old method : 0.00018644332885742188 length of segment : 26 time for calcul the mask position with numpy : 5.316734313964844e-05 nb_pixel_total : 861 time to create 1 rle with old method : 0.0016477108001708984 length of segment : 41 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -112.43828 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 9 time for calcul the mask position with numpy : 5.9604644775390625e-05 nb_pixel_total : 866 time to create 1 rle with old method : 0.0015804767608642578 length of segment : 36 time for calcul the mask position with numpy : 7.62939453125e-05 nb_pixel_total : 2330 time to create 1 rle with old method : 0.004026651382446289 length of segment : 53 time for calcul the mask position with numpy : 4.6253204345703125e-05 nb_pixel_total : 404 time to create 1 rle with old method : 0.0007355213165283203 length of segment : 26 time for calcul the mask position with numpy : 4.482269287109375e-05 nb_pixel_total : 355 time to create 1 rle with old method : 0.0006778240203857422 length of segment : 34 time for calcul the mask position with numpy : 0.00021505355834960938 nb_pixel_total : 6148 time to create 1 rle with old method : 0.010662317276000977 length of segment : 73 time for calcul the mask position with numpy : 0.00016260147094726562 nb_pixel_total : 6785 time to create 1 rle with old method : 0.011702537536621094 length of segment : 69 time for calcul the mask position with numpy : 5.125999450683594e-05 nb_pixel_total : 1110 time to create 1 rle with old method : 0.0013885498046875 length of segment : 29 time for calcul the mask position with numpy : 4.5299530029296875e-05 nb_pixel_total : 1179 time to create 1 rle with old method : 0.001430511474609375 length of segment : 34 time for calcul the mask position with numpy : 4.9591064453125e-05 nb_pixel_total : 944 time to create 1 rle with old method : 0.0012521743774414062 length of segment : 41 Processing 1 images image shape: (400, 400, 3) min: 3.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -98.80938 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 3 time for calcul the mask position with numpy : 0.0003154277801513672 nb_pixel_total : 20748 time to create 1 rle with old method : 0.02461719512939453 length of segment : 206 time for calcul the mask position with numpy : 8.893013000488281e-05 nb_pixel_total : 1973 time to create 1 rle with old method : 0.0025293827056884766 length of segment : 109 time for calcul the mask position with numpy : 9.131431579589844e-05 nb_pixel_total : 2241 time to create 1 rle with old method : 0.0027647018432617188 length of segment : 81 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -118.27813 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 6 time for calcul the mask position with numpy : 0.00013947486877441406 nb_pixel_total : 7012 time to create 1 rle with old method : 0.008690118789672852 length of segment : 133 time for calcul the mask position with numpy : 4.1484832763671875e-05 nb_pixel_total : 195 time to create 1 rle with old method : 0.0003540515899658203 length of segment : 25 time for calcul the mask position with numpy : 5.1021575927734375e-05 nb_pixel_total : 1129 time to create 1 rle with old method : 0.001398324966430664 length of segment : 62 time for calcul the mask position with numpy : 3.743171691894531e-05 nb_pixel_total : 436 time to create 1 rle with old method : 0.0005812644958496094 length of segment : 21 time for calcul the mask position with numpy : 0.0002694129943847656 nb_pixel_total : 7225 time to create 1 rle with old method : 0.009111166000366211 length of segment : 243 time for calcul the mask position with numpy : 5.8650970458984375e-05 nb_pixel_total : 194 time to create 1 rle with old method : 0.0003631114959716797 length of segment : 19 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 122.35859 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 6 time for calcul the mask position with numpy : 6.794929504394531e-05 nb_pixel_total : 1939 time to create 1 rle with old method : 0.0022389888763427734 length of segment : 147 time for calcul the mask position with numpy : 7.152557373046875e-05 nb_pixel_total : 592 time to create 1 rle with old method : 0.0013341903686523438 length of segment : 63 time for calcul the mask position with numpy : 3.3855438232421875e-05 nb_pixel_total : 206 time to create 1 rle with old method : 0.0003001689910888672 length of segment : 18 time for calcul the mask position with numpy : 3.5762786865234375e-05 nb_pixel_total : 357 time to create 1 rle with old method : 0.00047707557678222656 length of segment : 20 time for calcul the mask position with numpy : 4.458427429199219e-05 nb_pixel_total : 991 time to create 1 rle with old method : 0.0012476444244384766 length of segment : 33 time for calcul the mask position with numpy : 0.00010824203491210938 nb_pixel_total : 4176 time to create 1 rle with old method : 0.005298614501953125 length of segment : 66 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 206.00000 molded_images shape: (1, 640, 640, 3) min: -96.69688 max: 66.85859 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 0 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -115.96563 max: 130.94844 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 8 time for calcul the mask position with numpy : 0.00030684471130371094 nb_pixel_total : 24103 time to create 1 rle with old method : 0.02724313735961914 length of segment : 239 time for calcul the mask position with numpy : 7.05718994140625e-05 nb_pixel_total : 444 time to create 1 rle with old method : 0.0006060600280761719 length of segment : 23 time for calcul the mask position with numpy : 5.3882598876953125e-05 nb_pixel_total : 625 time to create 1 rle with old method : 0.0008227825164794922 length of segment : 35 time for calcul the mask position with numpy : 3.910064697265625e-05 nb_pixel_total : 231 time to create 1 rle with old method : 0.0003592967987060547 length of segment : 18 time for calcul the mask position with numpy : 3.504753112792969e-05 nb_pixel_total : 232 time to create 1 rle with old method : 0.0003616809844970703 length of segment : 17 time for calcul the mask position with numpy : 4.410743713378906e-05 nb_pixel_total : 625 time to create 1 rle with old method : 0.0008420944213867188 length of segment : 37 time for calcul the mask position with numpy : 4.9114227294921875e-05 nb_pixel_total : 182 time to create 1 rle with old method : 0.0002732276916503906 length of segment : 21 time for calcul the mask position with numpy : 4.029273986816406e-05 nb_pixel_total : 204 time to create 1 rle with old method : 0.0003485679626464844 length of segment : 23 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -103.18438 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 13 time for calcul the mask position with numpy : 0.00010418891906738281 nb_pixel_total : 2269 time to create 1 rle with old method : 0.004628658294677734 length of segment : 38 time for calcul the mask position with numpy : 5.793571472167969e-05 nb_pixel_total : 143 time to create 1 rle with old method : 0.0003712177276611328 length of segment : 24 time for calcul the mask position with numpy : 5.602836608886719e-05 nb_pixel_total : 305 time to create 1 rle with old method : 0.0006535053253173828 length of segment : 22 time for calcul the mask position with numpy : 0.00012135505676269531 nb_pixel_total : 1728 time to create 1 rle with old method : 0.0036554336547851562 length of segment : 105 time for calcul the mask position with numpy : 7.2479248046875e-05 nb_pixel_total : 618 time to create 1 rle with old method : 0.0012776851654052734 length of segment : 29 time for calcul the mask position with numpy : 6.604194641113281e-05 nb_pixel_total : 667 time to create 1 rle with old method : 0.0014679431915283203 length of segment : 57 time for calcul the mask position with numpy : 9.274482727050781e-05 nb_pixel_total : 1302 time to create 1 rle with old method : 0.00298309326171875 length of segment : 37 time for calcul the mask position with numpy : 4.1961669921875e-05 nb_pixel_total : 124 time to create 1 rle with old method : 0.0002644062042236328 length of segment : 13 time for calcul the mask position with numpy : 3.647804260253906e-05 nb_pixel_total : 94 time to create 1 rle with old method : 0.00019884109497070312 length of segment : 10 time for calcul the mask position with numpy : 3.504753112792969e-05 nb_pixel_total : 66 time to create 1 rle with old method : 0.00015425682067871094 length of segment : 8 time for calcul the mask position with numpy : 4.315376281738281e-05 nb_pixel_total : 369 time to create 1 rle with old method : 0.0006725788116455078 length of segment : 19 time for calcul the mask position with numpy : 4.744529724121094e-05 nb_pixel_total : 1025 time to create 1 rle with old method : 0.0013225078582763672 length of segment : 44 time for calcul the mask position with numpy : 6.079673767089844e-05 nb_pixel_total : 2203 time to create 1 rle with old method : 0.0029497146606445312 length of segment : 39 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -118.28203 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 6 time for calcul the mask position with numpy : 5.054473876953125e-05 nb_pixel_total : 512 time to create 1 rle with old method : 0.000667572021484375 length of segment : 53 time for calcul the mask position with numpy : 0.00015306472778320312 nb_pixel_total : 3196 time to create 1 rle with old method : 0.00422978401184082 length of segment : 125 time for calcul the mask position with numpy : 4.506111145019531e-05 nb_pixel_total : 229 time to create 1 rle with old method : 0.0003101825714111328 length of segment : 23 time for calcul the mask position with numpy : 0.00018525123596191406 nb_pixel_total : 1349 time to create 1 rle with old method : 0.0016946792602539062 length of segment : 111 time for calcul the mask position with numpy : 0.00022935867309570312 nb_pixel_total : 1602 time to create 1 rle with old method : 0.002298116683959961 length of segment : 58 time for calcul the mask position with numpy : 5.125999450683594e-05 nb_pixel_total : 900 time to create 1 rle with old method : 0.0011904239654541016 length of segment : 27 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 134.86016 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 8 time for calcul the mask position with numpy : 6.079673767089844e-05 nb_pixel_total : 393 time to create 1 rle with old method : 0.0006964206695556641 length of segment : 35 time for calcul the mask position with numpy : 0.00013971328735351562 nb_pixel_total : 912 time to create 1 rle with old method : 0.0017497539520263672 length of segment : 113 time for calcul the mask position with numpy : 5.91278076171875e-05 nb_pixel_total : 274 time to create 1 rle with old method : 0.0005466938018798828 length of segment : 30 time for calcul the mask position with numpy : 6.103515625e-05 nb_pixel_total : 477 time to create 1 rle with old method : 0.0009505748748779297 length of segment : 26 time for calcul the mask position with numpy : 4.9591064453125e-05 nb_pixel_total : 100 time to create 1 rle with old method : 0.00022792816162109375 length of segment : 17 time for calcul the mask position with numpy : 9.655952453613281e-05 nb_pixel_total : 1723 time to create 1 rle with old method : 0.0030090808868408203 length of segment : 85 time for calcul the mask position with numpy : 0.00010061264038085938 nb_pixel_total : 1760 time to create 1 rle with old method : 0.002992868423461914 length of segment : 89 time for calcul the mask position with numpy : 7.486343383789062e-05 nb_pixel_total : 773 time to create 1 rle with old method : 0.0015249252319335938 length of segment : 37 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 182.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 56.80547 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 1 time for calcul the mask position with numpy : 0.0009551048278808594 nb_pixel_total : 106551 time to create 1 rle with old method : 0.12018680572509766 length of segment : 279 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 148.61562 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 4 time for calcul the mask position with numpy : 0.00015544891357421875 nb_pixel_total : 1550 time to create 1 rle with old method : 0.0030956268310546875 length of segment : 52 time for calcul the mask position with numpy : 5.412101745605469e-05 nb_pixel_total : 393 time to create 1 rle with old method : 0.0007405281066894531 length of segment : 27 time for calcul the mask position with numpy : 0.0001475811004638672 nb_pixel_total : 5818 time to create 1 rle with old method : 0.007242679595947266 length of segment : 142 time for calcul the mask position with numpy : 3.62396240234375e-05 nb_pixel_total : 124 time to create 1 rle with old method : 0.0002014636993408203 length of segment : 11 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 143.43594 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 11 time for calcul the mask position with numpy : 9.036064147949219e-05 nb_pixel_total : 1094 time to create 1 rle with old method : 0.001531362533569336 length of segment : 59 time for calcul the mask position with numpy : 0.00012135505676269531 nb_pixel_total : 2235 time to create 1 rle with old method : 0.003103494644165039 length of segment : 47 time for calcul the mask position with numpy : 4.5299530029296875e-05 nb_pixel_total : 253 time to create 1 rle with old method : 0.0003845691680908203 length of segment : 27 time for calcul the mask position with numpy : 3.933906555175781e-05 nb_pixel_total : 319 time to create 1 rle with old method : 0.00041413307189941406 length of segment : 29 time for calcul the mask position with numpy : 3.218650817871094e-05 nb_pixel_total : 195 time to create 1 rle with old method : 0.0003046989440917969 length of segment : 11 time for calcul the mask position with numpy : 4.363059997558594e-05 nb_pixel_total : 518 time to create 1 rle with old method : 0.0007295608520507812 length of segment : 29 time for calcul the mask position with numpy : 0.00010824203491210938 nb_pixel_total : 3384 time to create 1 rle with old method : 0.004294395446777344 length of segment : 64 time for calcul the mask position with numpy : 5.7220458984375e-05 nb_pixel_total : 726 time to create 1 rle with old method : 0.0009391307830810547 length of segment : 35 time for calcul the mask position with numpy : 3.24249267578125e-05 nb_pixel_total : 138 time to create 1 rle with old method : 0.00023651123046875 length of segment : 11 time for calcul the mask position with numpy : 3.7670135498046875e-05 nb_pixel_total : 356 time to create 1 rle with old method : 0.0006003379821777344 length of segment : 18 time for calcul the mask position with numpy : 4.863739013671875e-05 nb_pixel_total : 810 time to create 1 rle with old method : 0.0010526180267333984 length of segment : 36 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 149.86562 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 10 time for calcul the mask position with numpy : 6.175041198730469e-05 nb_pixel_total : 515 time to create 1 rle with old method : 0.0007610321044921875 length of segment : 37 time for calcul the mask position with numpy : 3.838539123535156e-05 nb_pixel_total : 347 time to create 1 rle with old method : 0.0004878044128417969 length of segment : 21 time for calcul the mask position with numpy : 7.319450378417969e-05 nb_pixel_total : 4043 time to create 1 rle with old method : 0.004967689514160156 length of segment : 62 time for calcul the mask position with numpy : 6.0558319091796875e-05 nb_pixel_total : 628 time to create 1 rle with old method : 0.0008382797241210938 length of segment : 55 time for calcul the mask position with numpy : 4.220008850097656e-05 nb_pixel_total : 105 time to create 1 rle with old method : 0.00021529197692871094 length of segment : 27 time for calcul the mask position with numpy : 3.218650817871094e-05 nb_pixel_total : 247 time to create 1 rle with old method : 0.0003528594970703125 length of segment : 14 time for calcul the mask position with numpy : 4.100799560546875e-05 nb_pixel_total : 151 time to create 1 rle with old method : 0.0002644062042236328 length of segment : 36 time for calcul the mask position with numpy : 4.744529724121094e-05 nb_pixel_total : 836 time to create 1 rle with old method : 0.0010793209075927734 length of segment : 58 time for calcul the mask position with numpy : 4.410743713378906e-05 nb_pixel_total : 917 time to create 1 rle with old method : 0.0011746883392333984 length of segment : 62 time for calcul the mask position with numpy : 4.291534423828125e-05 nb_pixel_total : 276 time to create 1 rle with old method : 0.0005090236663818359 length of segment : 31 Processing 1 images image shape: (280, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 132.26250 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 0 Detection mask done ! Trying to reset tf kernel 4170215 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 4465 tf kernel not reseted sub process len(results) : 194 len(list_Values) 194 None max_time_sub_proc : 3600 parent process len(results) : 0 len(list_Values) 194 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 9754 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Catched exception ! Connect or reconnect ! Number saved : None batch 1 Loaded 194 chid ids of type : 4228 Number RLEs to save : 10986 save missing photos in datou_result : time spend for datou_step_exec : 23.772672653198242 time spend to save output : 0.8966152667999268 total time spend for step 1 : 24.66928791999817 step2:brightness Tue Feb 4 14:20:53 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure inside step calcul brightness treat image : temp/1738675228_4169668_1334557442_d5f1219667f9235b10e82a936d960a30.jpg treat image : temp/1738675228_4169668_1334557361_dbb588789b7d6662279013b5cda70712.jpg Inside saveOutput : final : False verbose : 0 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 2 time used for this insertion : 0.012440919876098633 begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 2 time used for this insertion : 0.011349678039550781 save missing photos in datou_result : time spend for datou_step_exec : 0.5500671863555908 time spend to save output : 0.02816915512084961 total time spend for step 2 : 0.5782363414764404 step3:blur_detection Tue Feb 4 14:20:54 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure inside step blur_detection methode: ratio et variance treat image : temp/1738675228_4169668_1334557442_d5f1219667f9235b10e82a936d960a30.jpg resize: (1080, 1920) 1334557442 -7.720226752089334 treat image : temp/1738675228_4169668_1334557361_dbb588789b7d6662279013b5cda70712.jpg resize: (1080, 1920) 1334557361 -7.688103685602205 Inside saveOutput : final : False verbose : 0 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 2 time used for this insertion : 0.012550115585327148 begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 2 time used for this insertion : 0.010955810546875 save missing photos in datou_result : time spend for datou_step_exec : 1.6178386211395264 time spend to save output : 0.028356075286865234 total time spend for step 3 : 1.6461946964263916 step4:rle_unique_nms_with_priority Tue Feb 4 14:20:55 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms batch 1 Loaded 194 chid ids of type : 4228 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++nb_obj : 98 nb_hashtags : 6 time to prepare the origin masks : 1.2126305103302002 time for calcul the mask position with numpy : 0.027234792709350586 nb_pixel_total : 1737029 time to create 1 rle with new method : 0.05479145050048828 time for calcul the mask position with numpy : 0.008775949478149414 nb_pixel_total : 1774 time to create 1 rle with old method : 0.002083301544189453 time for calcul the mask position with numpy : 0.008554935455322266 nb_pixel_total : 19 time to create 1 rle with old method : 5.125999450683594e-05 time for calcul the mask position with numpy : 0.00849604606628418 nb_pixel_total : 70 time to create 1 rle with old method : 0.0001232624053955078 time for calcul the mask position with numpy : 0.008491992950439453 nb_pixel_total : 83 time to create 1 rle with old method : 0.00010657310485839844 time for calcul the mask position with numpy : 0.008560419082641602 nb_pixel_total : 495 time to create 1 rle with old method : 0.0007107257843017578 time for calcul the mask position with numpy : 0.008482217788696289 nb_pixel_total : 387 time to create 1 rle with old method : 0.00044465065002441406 time for calcul the mask position with numpy : 0.008518695831298828 nb_pixel_total : 855 time to create 1 rle with old method : 0.0009868144989013672 time for calcul the mask position with numpy : 0.00846719741821289 nb_pixel_total : 812 time to create 1 rle with old method : 0.0009517669677734375 time for calcul the mask position with numpy : 0.008511781692504883 nb_pixel_total : 429 time to create 1 rle with old method : 0.0004918575286865234 time for calcul the mask position with numpy : 0.008541584014892578 nb_pixel_total : 18 time to create 1 rle with old method : 6.556510925292969e-05 time for calcul the mask position with numpy : 0.008504390716552734 nb_pixel_total : 1738 time to create 1 rle with old method : 0.001985311508178711 time for calcul the mask position with numpy : 0.008507966995239258 nb_pixel_total : 56 time to create 1 rle with old method : 0.00010442733764648438 time for calcul the mask position with numpy : 0.008890867233276367 nb_pixel_total : 410 time to create 1 rle with old method : 0.00048232078552246094 time for calcul the mask position with numpy : 0.008511543273925781 nb_pixel_total : 82 time to create 1 rle with old method : 0.00010609626770019531 time for calcul the mask position with numpy : 0.008505582809448242 nb_pixel_total : 2614 time to create 1 rle with old method : 0.00296783447265625 time for calcul the mask position with numpy : 0.00853729248046875 nb_pixel_total : 42 time to create 1 rle with old method : 6.580352783203125e-05 time for calcul the mask position with numpy : 0.008527517318725586 nb_pixel_total : 762 time to create 1 rle with old method : 0.0008847713470458984 time for calcul the mask position with numpy : 0.009992599487304688 nb_pixel_total : 48664 time to create 1 rle with old method : 0.06354904174804688 time for calcul the mask position with numpy : 0.009826898574829102 nb_pixel_total : 1133 time to create 1 rle with old method : 0.0020284652709960938 time for calcul the mask position with numpy : 0.00977015495300293 nb_pixel_total : 7197 time to create 1 rle with old method : 0.011754274368286133 time for calcul the mask position with numpy : 0.009684324264526367 nb_pixel_total : 1186 time to create 1 rle with old method : 0.0019974708557128906 time for calcul the mask position with numpy : 0.009426116943359375 nb_pixel_total : 112 time to create 1 rle with old method : 0.00014543533325195312 time for calcul the mask position with numpy : 0.008501529693603516 nb_pixel_total : 93 time to create 1 rle with old method : 0.00012493133544921875 time for calcul the mask position with numpy : 0.00854635238647461 nb_pixel_total : 1432 time to create 1 rle with old method : 0.001668691635131836 time for calcul the mask position with numpy : 0.008493185043334961 nb_pixel_total : 916 time to create 1 rle with old method : 0.0010428428649902344 time for calcul the mask position with numpy : 0.008501291275024414 nb_pixel_total : 181 time to create 1 rle with old method : 0.0002219676971435547 time for calcul the mask position with numpy : 0.008483409881591797 nb_pixel_total : 2592 time to create 1 rle with old method : 0.0030117034912109375 time for calcul the mask position with numpy : 0.008522748947143555 nb_pixel_total : 2655 time to create 1 rle with old method : 0.00307464599609375 time for calcul the mask position with numpy : 0.008551359176635742 nb_pixel_total : 168 time to create 1 rle with old method : 0.0002605915069580078 time for calcul the mask position with numpy : 0.009150266647338867 nb_pixel_total : 89 time to create 1 rle with old method : 0.00014853477478027344 time for calcul the mask position with numpy : 0.008592605590820312 nb_pixel_total : 5623 time to create 1 rle with old method : 0.006436824798583984 time for calcul the mask position with numpy : 0.008959054946899414 nb_pixel_total : 225 time to create 1 rle with old method : 0.0002961158752441406 time for calcul the mask position with numpy : 0.008757591247558594 nb_pixel_total : 182 time to create 1 rle with old method : 0.0003142356872558594 time for calcul the mask position with numpy : 0.008738517761230469 nb_pixel_total : 1023 time to create 1 rle with old method : 0.001220703125 time for calcul the mask position with numpy : 0.008741140365600586 nb_pixel_total : 356 time to create 1 rle with old method : 0.0004515647888183594 time for calcul the mask position with numpy : 0.008846044540405273 nb_pixel_total : 22 time to create 1 rle with old method : 7.915496826171875e-05 time for calcul the mask position with numpy : 0.008869171142578125 nb_pixel_total : 76 time to create 1 rle with old method : 0.00012826919555664062 time for calcul the mask position with numpy : 0.00884246826171875 nb_pixel_total : 3656 time to create 1 rle with old method : 0.004080295562744141 time for calcul the mask position with numpy : 0.008846759796142578 nb_pixel_total : 216 time to create 1 rle with old method : 0.0003619194030761719 time for calcul the mask position with numpy : 0.008802175521850586 nb_pixel_total : 2186 time to create 1 rle with old method : 0.002606630325317383 time for calcul the mask position with numpy : 0.011336326599121094 nb_pixel_total : 431 time to create 1 rle with old method : 0.0009286403656005859 time for calcul the mask position with numpy : 0.013457298278808594 nb_pixel_total : 616 time to create 1 rle with old method : 0.0007538795471191406 time for calcul the mask position with numpy : 0.008680343627929688 nb_pixel_total : 3264 time to create 1 rle with old method : 0.003985881805419922 time for calcul the mask position with numpy : 0.008948802947998047 nb_pixel_total : 61 time to create 1 rle with old method : 0.000331878662109375 time for calcul the mask position with numpy : 0.008752107620239258 nb_pixel_total : 1052 time to create 1 rle with old method : 0.0012791156768798828 time for calcul the mask position with numpy : 0.00953531265258789 nb_pixel_total : 198 time to create 1 rle with old method : 0.0002663135528564453 time for calcul the mask position with numpy : 0.010351181030273438 nb_pixel_total : 30553 time to create 1 rle with old method : 0.03274822235107422 time for calcul the mask position with numpy : 0.008268117904663086 nb_pixel_total : 124 time to create 1 rle with old method : 0.00016045570373535156 time for calcul the mask position with numpy : 0.00836038589477539 nb_pixel_total : 22 time to create 1 rle with old method : 4.863739013671875e-05 time for calcul the mask position with numpy : 0.008306026458740234 nb_pixel_total : 286 time to create 1 rle with old method : 0.0003161430358886719 time for calcul the mask position with numpy : 0.008358001708984375 nb_pixel_total : 366 time to create 1 rle with old method : 0.0004425048828125 time for calcul the mask position with numpy : 0.008345603942871094 nb_pixel_total : 10832 time to create 1 rle with old method : 0.011277914047241211 time for calcul the mask position with numpy : 0.008311271667480469 nb_pixel_total : 869 time to create 1 rle with old method : 0.0010848045349121094 time for calcul the mask position with numpy : 0.008160114288330078 nb_pixel_total : 1318 time to create 1 rle with old method : 0.0015325546264648438 time for calcul the mask position with numpy : 0.008188009262084961 nb_pixel_total : 448 time to create 1 rle with old method : 0.0005888938903808594 time for calcul the mask position with numpy : 0.008294105529785156 nb_pixel_total : 3812 time to create 1 rle with old method : 0.004165172576904297 time for calcul the mask position with numpy : 0.008184194564819336 nb_pixel_total : 4494 time to create 1 rle with old method : 0.005023956298828125 time for calcul the mask position with numpy : 0.008177518844604492 nb_pixel_total : 101 time to create 1 rle with old method : 0.00012731552124023438 time for calcul the mask position with numpy : 0.008133649826049805 nb_pixel_total : 453 time to create 1 rle with old method : 0.0005586147308349609 time for calcul the mask position with numpy : 0.008042097091674805 nb_pixel_total : 437 time to create 1 rle with old method : 0.0004487037658691406 time for calcul the mask position with numpy : 0.008486032485961914 nb_pixel_total : 389 time to create 1 rle with old method : 0.0005657672882080078 time for calcul the mask position with numpy : 0.009816884994506836 nb_pixel_total : 1391 time to create 1 rle with old method : 0.0017094612121582031 time for calcul the mask position with numpy : 0.008220672607421875 nb_pixel_total : 649 time to create 1 rle with old method : 0.0007767677307128906 time for calcul the mask position with numpy : 0.010686635971069336 nb_pixel_total : 73 time to create 1 rle with old method : 0.00014662742614746094 time for calcul the mask position with numpy : 0.008333921432495117 nb_pixel_total : 1867 time to create 1 rle with old method : 0.0021452903747558594 time for calcul the mask position with numpy : 0.008268356323242188 nb_pixel_total : 133 time to create 1 rle with old method : 0.00018262863159179688 time for calcul the mask position with numpy : 0.008363008499145508 nb_pixel_total : 152 time to create 1 rle with old method : 0.00018596649169921875 time for calcul the mask position with numpy : 0.008484601974487305 nb_pixel_total : 3329 time to create 1 rle with old method : 0.0036983489990234375 time for calcul the mask position with numpy : 0.009242534637451172 nb_pixel_total : 1895 time to create 1 rle with old method : 0.0033338069915771484 time for calcul the mask position with numpy : 0.010796070098876953 nb_pixel_total : 3 time to create 1 rle with old method : 5.1021575927734375e-05 time for calcul the mask position with numpy : 0.008781909942626953 nb_pixel_total : 249 time to create 1 rle with old method : 0.00031375885009765625 time for calcul the mask position with numpy : 0.009240388870239258 nb_pixel_total : 106476 time to create 1 rle with old method : 0.12590527534484863 time for calcul the mask position with numpy : 0.008520126342773438 nb_pixel_total : 174 time to create 1 rle with old method : 0.00023436546325683594 time for calcul the mask position with numpy : 0.008458375930786133 nb_pixel_total : 3175 time to create 1 rle with old method : 0.003669261932373047 time for calcul the mask position with numpy : 0.008616924285888672 nb_pixel_total : 607 time to create 1 rle with old method : 0.0007712841033935547 time for calcul the mask position with numpy : 0.009902477264404297 nb_pixel_total : 50599 time to create 1 rle with old method : 0.05585503578186035 time for calcul the mask position with numpy : 0.008517980575561523 nb_pixel_total : 351 time to create 1 rle with old method : 0.00041866302490234375 time for calcul the mask position with numpy : 0.008441448211669922 nb_pixel_total : 101 time to create 1 rle with old method : 0.00016355514526367188 time for calcul the mask position with numpy : 0.008533954620361328 nb_pixel_total : 6 time to create 1 rle with old method : 2.4080276489257812e-05 time for calcul the mask position with numpy : 0.008415937423706055 nb_pixel_total : 570 time to create 1 rle with old method : 0.0006616115570068359 time for calcul the mask position with numpy : 0.008502483367919922 nb_pixel_total : 1490 time to create 1 rle with old method : 0.0017192363739013672 time for calcul the mask position with numpy : 0.008591175079345703 nb_pixel_total : 244 time to create 1 rle with old method : 0.0002875328063964844 time for calcul the mask position with numpy : 0.00854635238647461 nb_pixel_total : 115 time to create 1 rle with old method : 0.0001437664031982422 time for calcul the mask position with numpy : 0.008484363555908203 nb_pixel_total : 1974 time to create 1 rle with old method : 0.002280712127685547 time for calcul the mask position with numpy : 0.008653640747070312 nb_pixel_total : 778 time to create 1 rle with old method : 0.0009200572967529297 time for calcul the mask position with numpy : 0.008567571640014648 nb_pixel_total : 458 time to create 1 rle with old method : 0.0005867481231689453 time for calcul the mask position with numpy : 0.00861048698425293 nb_pixel_total : 1006 time to create 1 rle with old method : 0.0012316703796386719 time for calcul the mask position with numpy : 0.00852060317993164 nb_pixel_total : 352 time to create 1 rle with old method : 0.00046372413635253906 time for calcul the mask position with numpy : 0.008554697036743164 nb_pixel_total : 332 time to create 1 rle with old method : 0.0003979206085205078 time for calcul the mask position with numpy : 0.008585214614868164 nb_pixel_total : 428 time to create 1 rle with old method : 0.0005409717559814453 time for calcul the mask position with numpy : 0.00851130485534668 nb_pixel_total : 1183 time to create 1 rle with old method : 0.0013968944549560547 time for calcul the mask position with numpy : 0.00847768783569336 nb_pixel_total : 294 time to create 1 rle with old method : 0.0003809928894042969 time for calcul the mask position with numpy : 0.008818864822387695 nb_pixel_total : 10 time to create 1 rle with old method : 4.220008850097656e-05 time for calcul the mask position with numpy : 0.008958816528320312 nb_pixel_total : 1031 time to create 1 rle with old method : 0.0012562274932861328 time for calcul the mask position with numpy : 0.008476972579956055 nb_pixel_total : 984 time to create 1 rle with old method : 0.001155853271484375 time for calcul the mask position with numpy : 0.008572816848754883 nb_pixel_total : 14 time to create 1 rle with old method : 3.4809112548828125e-05 time for calcul the mask position with numpy : 0.008643150329589844 nb_pixel_total : 3321 time to create 1 rle with old method : 0.005383729934692383 time for calcul the mask position with numpy : 0.00984954833984375 nb_pixel_total : 2 time to create 1 rle with old method : 3.361701965332031e-05 create new chi : 1.3561813831329346 after preparing all the mask , begin to delete the rle from the crop_hashtag_id => VR 28-11-20 : il faut déplacer cela apres le save_crop_hashtag_ids_obj de la ligne 9514 ! we have 0 chi objets contains the rles time to delete rle : 0.017328262329101562 batch 1 Loaded 99 chid ids of type : 4230 Number RLEs to save : 11670 TO DO : save crop sub photo not yet done ! save time : 0.7059371471405029 nb_obj : 92 nb_hashtags : 6 time to prepare the origin masks : 2.1906075477600098 time for calcul the mask position with numpy : 0.04585599899291992 nb_pixel_total : 1831934 time to create 1 rle with new method : 0.07738757133483887 time for calcul the mask position with numpy : 0.008244514465332031 nb_pixel_total : 1939 time to create 1 rle with old method : 0.002279520034790039 time for calcul the mask position with numpy : 0.007806539535522461 nb_pixel_total : 20748 time to create 1 rle with old method : 0.02275538444519043 time for calcul the mask position with numpy : 0.007784605026245117 nb_pixel_total : 86 time to create 1 rle with old method : 0.00015974044799804688 time for calcul the mask position with numpy : 0.007769346237182617 nb_pixel_total : 13 time to create 1 rle with old method : 5.125999450683594e-05 time for calcul the mask position with numpy : 0.008232593536376953 nb_pixel_total : 1221 time to create 1 rle with old method : 0.0014963150024414062 time for calcul the mask position with numpy : 0.007518291473388672 nb_pixel_total : 355 time to create 1 rle with old method : 0.0004584789276123047 time for calcul the mask position with numpy : 0.006712675094604492 nb_pixel_total : 1110 time to create 1 rle with old method : 0.0013272762298583984 time for calcul the mask position with numpy : 0.011807441711425781 nb_pixel_total : 106 time to create 1 rle with old method : 0.00017070770263671875 time for calcul the mask position with numpy : 0.012185335159301758 nb_pixel_total : 83 time to create 1 rle with old method : 0.00016117095947265625 time for calcul the mask position with numpy : 0.012219667434692383 nb_pixel_total : 95 time to create 1 rle with old method : 0.00015854835510253906 time for calcul the mask position with numpy : 0.012200355529785156 nb_pixel_total : 411 time to create 1 rle with old method : 0.0005135536193847656 time for calcul the mask position with numpy : 0.01641249656677246 nb_pixel_total : 736 time to create 1 rle with old method : 0.0009009838104248047 time for calcul the mask position with numpy : 0.01134347915649414 nb_pixel_total : 404 time to create 1 rle with old method : 0.00048613548278808594 time for calcul the mask position with numpy : 0.015599727630615234 nb_pixel_total : 2330 time to create 1 rle with old method : 0.005873441696166992 time for calcul the mask position with numpy : 0.017473220825195312 nb_pixel_total : 7225 time to create 1 rle with old method : 0.014737606048583984 time for calcul the mask position with numpy : 0.011562347412109375 nb_pixel_total : 866 time to create 1 rle with old method : 0.0010056495666503906 time for calcul the mask position with numpy : 0.010250091552734375 nb_pixel_total : 6148 time to create 1 rle with old method : 0.006934642791748047 time for calcul the mask position with numpy : 0.010413646697998047 nb_pixel_total : 1314 time to create 1 rle with old method : 0.0016274452209472656 time for calcul the mask position with numpy : 0.01015782356262207 nb_pixel_total : 551 time to create 1 rle with old method : 0.0006170272827148438 time for calcul the mask position with numpy : 0.010479450225830078 nb_pixel_total : 1129 time to create 1 rle with old method : 0.0012536048889160156 time for calcul the mask position with numpy : 0.006500720977783203 nb_pixel_total : 195 time to create 1 rle with old method : 0.0002601146697998047 time for calcul the mask position with numpy : 0.006676435470581055 nb_pixel_total : 944 time to create 1 rle with old method : 0.0011234283447265625 time for calcul the mask position with numpy : 0.006304264068603516 nb_pixel_total : 194 time to create 1 rle with old method : 0.00024175643920898438 time for calcul the mask position with numpy : 0.006338357925415039 nb_pixel_total : 4176 time to create 1 rle with old method : 0.0046575069427490234 time for calcul the mask position with numpy : 0.0063741207122802734 nb_pixel_total : 676 time to create 1 rle with old method : 0.0008494853973388672 time for calcul the mask position with numpy : 0.006058692932128906 nb_pixel_total : 1973 time to create 1 rle with old method : 0.002392292022705078 time for calcul the mask position with numpy : 0.006345510482788086 nb_pixel_total : 7012 time to create 1 rle with old method : 0.00833439826965332 time for calcul the mask position with numpy : 0.006743907928466797 nb_pixel_total : 206 time to create 1 rle with old method : 0.00026702880859375 time for calcul the mask position with numpy : 0.006940603256225586 nb_pixel_total : 436 time to create 1 rle with old method : 0.0005242824554443359 time for calcul the mask position with numpy : 0.007348060607910156 nb_pixel_total : 991 time to create 1 rle with old method : 0.0011725425720214844 time for calcul the mask position with numpy : 0.007112741470336914 nb_pixel_total : 357 time to create 1 rle with old method : 0.0004413127899169922 time for calcul the mask position with numpy : 0.007387876510620117 nb_pixel_total : 667 time to create 1 rle with old method : 0.0008211135864257812 time for calcul the mask position with numpy : 0.009088993072509766 nb_pixel_total : 900 time to create 1 rle with old method : 0.0010614395141601562 time for calcul the mask position with numpy : 0.012911081314086914 nb_pixel_total : 216 time to create 1 rle with old method : 0.00039196014404296875 time for calcul the mask position with numpy : 0.013879537582397461 nb_pixel_total : 94 time to create 1 rle with old method : 0.00018835067749023438 time for calcul the mask position with numpy : 0.013528108596801758 nb_pixel_total : 18 time to create 1 rle with old method : 0.00013184547424316406 time for calcul the mask position with numpy : 0.013524770736694336 nb_pixel_total : 2269 time to create 1 rle with old method : 0.003732442855834961 time for calcul the mask position with numpy : 0.013795137405395508 nb_pixel_total : 618 time to create 1 rle with old method : 0.0010523796081542969 time for calcul the mask position with numpy : 0.01321864128112793 nb_pixel_total : 3196 time to create 1 rle with old method : 0.0037238597869873047 time for calcul the mask position with numpy : 0.012270689010620117 nb_pixel_total : 66 time to create 1 rle with old method : 0.00010704994201660156 time for calcul the mask position with numpy : 0.012307167053222656 nb_pixel_total : 1602 time to create 1 rle with old method : 0.001972198486328125 time for calcul the mask position with numpy : 0.012368917465209961 nb_pixel_total : 1302 time to create 1 rle with old method : 0.001558065414428711 time for calcul the mask position with numpy : 0.0125274658203125 nb_pixel_total : 36 time to create 1 rle with old method : 8.845329284667969e-05 time for calcul the mask position with numpy : 0.012561798095703125 nb_pixel_total : 19 time to create 1 rle with old method : 6.127357482910156e-05 time for calcul the mask position with numpy : 0.012693643569946289 nb_pixel_total : 625 time to create 1 rle with old method : 0.0007414817810058594 time for calcul the mask position with numpy : 0.0128631591796875 nb_pixel_total : 231 time to create 1 rle with old method : 0.00033164024353027344 time for calcul the mask position with numpy : 0.014160633087158203 nb_pixel_total : 274 time to create 1 rle with old method : 0.0003571510314941406 time for calcul the mask position with numpy : 0.013985872268676758 nb_pixel_total : 369 time to create 1 rle with old method : 0.0006496906280517578 time for calcul the mask position with numpy : 0.008076190948486328 nb_pixel_total : 24103 time to create 1 rle with old method : 0.02662372589111328 time for calcul the mask position with numpy : 0.008107662200927734 nb_pixel_total : 305 time to create 1 rle with old method : 0.0003807544708251953 time for calcul the mask position with numpy : 0.008344650268554688 nb_pixel_total : 652 time to create 1 rle with old method : 0.0009453296661376953 time for calcul the mask position with numpy : 0.0077207088470458984 nb_pixel_total : 912 time to create 1 rle with old method : 0.0011565685272216797 time for calcul the mask position with numpy : 0.008353471755981445 nb_pixel_total : 100 time to create 1 rle with old method : 0.0001461505889892578 time for calcul the mask position with numpy : 0.008163928985595703 nb_pixel_total : 424 time to create 1 rle with old method : 0.000644683837890625 time for calcul the mask position with numpy : 0.008095264434814453 nb_pixel_total : 1349 time to create 1 rle with old method : 0.002138853073120117 time for calcul the mask position with numpy : 0.008387327194213867 nb_pixel_total : 512 time to create 1 rle with old method : 0.0006608963012695312 time for calcul the mask position with numpy : 0.008274555206298828 nb_pixel_total : 444 time to create 1 rle with old method : 0.0005667209625244141 time for calcul the mask position with numpy : 0.008406877517700195 nb_pixel_total : 393 time to create 1 rle with old method : 0.00046539306640625 time for calcul the mask position with numpy : 0.008122444152832031 nb_pixel_total : 232 time to create 1 rle with old method : 0.0003082752227783203 time for calcul the mask position with numpy : 0.008424758911132812 nb_pixel_total : 1723 time to create 1 rle with old method : 0.0020360946655273438 time for calcul the mask position with numpy : 0.007942914962768555 nb_pixel_total : 100 time to create 1 rle with old method : 0.00017952919006347656 time for calcul the mask position with numpy : 0.008256196975708008 nb_pixel_total : 124 time to create 1 rle with old method : 0.0001811981201171875 time for calcul the mask position with numpy : 0.008230924606323242 nb_pixel_total : 26 time to create 1 rle with old method : 7.915496826171875e-05 time for calcul the mask position with numpy : 0.008737325668334961 nb_pixel_total : 182 time to create 1 rle with old method : 0.00024700164794921875 time for calcul the mask position with numpy : 0.008144617080688477 nb_pixel_total : 143 time to create 1 rle with old method : 0.0002040863037109375 time for calcul the mask position with numpy : 0.0083465576171875 nb_pixel_total : 1728 time to create 1 rle with old method : 0.001973867416381836 time for calcul the mask position with numpy : 0.009390592575073242 nb_pixel_total : 106551 time to create 1 rle with old method : 0.11583113670349121 time for calcul the mask position with numpy : 0.007675647735595703 nb_pixel_total : 195 time to create 1 rle with old method : 0.00027251243591308594 time for calcul the mask position with numpy : 0.007856130599975586 nb_pixel_total : 4043 time to create 1 rle with old method : 0.0059795379638671875 time for calcul the mask position with numpy : 0.008028268814086914 nb_pixel_total : 138 time to create 1 rle with old method : 0.00020742416381835938 time for calcul the mask position with numpy : 0.007832050323486328 nb_pixel_total : 518 time to create 1 rle with old method : 0.0007152557373046875 time for calcul the mask position with numpy : 0.007954597473144531 nb_pixel_total : 393 time to create 1 rle with old method : 0.0005517005920410156 time for calcul the mask position with numpy : 0.008360624313354492 nb_pixel_total : 356 time to create 1 rle with old method : 0.00047969818115234375 time for calcul the mask position with numpy : 0.007787227630615234 nb_pixel_total : 1550 time to create 1 rle with old method : 0.0019073486328125 time for calcul the mask position with numpy : 0.008075475692749023 nb_pixel_total : 810 time to create 1 rle with old method : 0.0010099411010742188 time for calcul the mask position with numpy : 0.008095026016235352 nb_pixel_total : 247 time to create 1 rle with old method : 0.00032782554626464844 time for calcul the mask position with numpy : 0.007917642593383789 nb_pixel_total : 124 time to create 1 rle with old method : 0.00018095970153808594 time for calcul the mask position with numpy : 0.007975339889526367 nb_pixel_total : 5818 time to create 1 rle with old method : 0.006788492202758789 time for calcul the mask position with numpy : 0.008387088775634766 nb_pixel_total : 2235 time to create 1 rle with old method : 0.003811359405517578 time for calcul the mask position with numpy : 0.00815272331237793 nb_pixel_total : 253 time to create 1 rle with old method : 0.0004482269287109375 time for calcul the mask position with numpy : 0.008142232894897461 nb_pixel_total : 515 time to create 1 rle with old method : 0.0009012222290039062 time for calcul the mask position with numpy : 0.008014440536499023 nb_pixel_total : 726 time to create 1 rle with old method : 0.0012390613555908203 time for calcul the mask position with numpy : 0.00837254524230957 nb_pixel_total : 319 time to create 1 rle with old method : 0.0005407333374023438 time for calcul the mask position with numpy : 0.008051633834838867 nb_pixel_total : 347 time to create 1 rle with old method : 0.0005869865417480469 time for calcul the mask position with numpy : 0.008376836776733398 nb_pixel_total : 1094 time to create 1 rle with old method : 0.0018129348754882812 time for calcul the mask position with numpy : 0.00922250747680664 nb_pixel_total : 139 time to create 1 rle with old method : 0.0002789497375488281 time for calcul the mask position with numpy : 0.008459806442260742 nb_pixel_total : 262 time to create 1 rle with old method : 0.0004787445068359375 time for calcul the mask position with numpy : 0.008355379104614258 nb_pixel_total : 96 time to create 1 rle with old method : 0.00015211105346679688 time for calcul the mask position with numpy : 0.007798910140991211 nb_pixel_total : 836 time to create 1 rle with old method : 0.0010075569152832031 time for calcul the mask position with numpy : 0.00742650032043457 nb_pixel_total : 105 time to create 1 rle with old method : 0.00016498565673828125 time for calcul the mask position with numpy : 0.007376909255981445 nb_pixel_total : 628 time to create 1 rle with old method : 0.0007612705230712891 time for calcul the mask position with numpy : 0.010758161544799805 nb_pixel_total : 3384 time to create 1 rle with old method : 0.0038862228393554688 create new chi : 1.2813010215759277 after preparing all the mask , begin to delete the rle from the crop_hashtag_id => VR 28-11-20 : il faut déplacer cela apres le save_crop_hashtag_ids_obj de la ligne 9514 ! we have 0 chi objets contains the rles time to delete rle : 0.0016970634460449219 batch 1 Loaded 93 chid ids of type : 4230 Number RLEs to save : 10020 TO DO : save crop sub photo not yet done ! save time : 0.5975503921508789 map_output_result : {1334557442: (0.0, 'Should be the crop_list due to order', 0.0), 1334557361: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1334557442, 1334557361] Looping around the photos to save general results len do output : 2 /1334557442.Didn't retrieve data . /1334557361.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2545063') ('4234', '20210830', '1334557442', None, None, None, None, None, '2545063') ('4234', None, None, None, None, None, None, None, '2545063') ('4234', '20210830', '1334557361', None, None, None, None, None, '2545063') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 6 time used for this insertion : 0.014626502990722656 save_final save missing photos in datou_result : time spend for datou_step_exec : 7.775306224822998 time spend to save output : 0.015074491500854492 total time spend for step 4 : 7.7903807163238525 step5:crop_condition Tue Feb 4 14:21:03 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 4230 Loading chi in step crop for list_pids : 2 ! batch 1 Loaded 192 chid ids of type : 4230 begin to crop the class : papier param for this class : {'min_score': 0.6} filtre for class : papier hashtag_id of this class : 492668766 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 48 About to insert : list_path_to_insert length 48 new photo from crops ! About to upload 48 photos upload in portfolio : 4869462 init cache_photo without model_param we have 48 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738675265_4169668 we have uploaded 48 photos in the portfolio 4869462 time of upload the photos Elapsed time : 10.935394525527954 we have finished the crop for the class : papier begin to crop the class : carton param for this class : {'min_score': 0.6} filtre for class : carton hashtag_id of this class : 492774966 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 28 About to insert : list_path_to_insert length 28 new photo from crops ! About to upload 28 photos upload in portfolio : 4869462 init cache_photo without model_param we have 28 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738675276_4169668 we have uploaded 28 photos in the portfolio 4869462 time of upload the photos Elapsed time : 6.570641756057739 we have finished the crop for the class : carton begin to crop the class : metal param for this class : {'min_score': 0.6} filtre for class : metal hashtag_id of this class : 492628673 begin to crop the class : pet_clair param for this class : {'min_score': 0.6} filtre for class : pet_clair hashtag_id of this class : 2107755846 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 12 About to insert : list_path_to_insert length 12 new photo from crops ! About to upload 12 photos upload in portfolio : 4869462 init cache_photo without model_param we have 12 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738675283_4169668 we have uploaded 12 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.8628053665161133 we have finished the crop for the class : pet_clair begin to crop the class : autre param for this class : {'min_score': 0.6} filtre for class : autre hashtag_id of this class : 494826614 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 28 About to insert : list_path_to_insert length 28 new photo from crops ! About to upload 28 photos upload in portfolio : 4869462 init cache_photo without model_param we have 28 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738675287_4169668 we have uploaded 28 photos in the portfolio 4869462 time of upload the photos Elapsed time : 5.822359323501587 we have finished the crop for the class : autre begin to crop the class : pehd param for this class : {'min_score': 0.6} filtre for class : pehd hashtag_id of this class : 628944319 begin to crop the class : pet_fonce param for this class : {'min_score': 0.6} filtre for class : pet_fonce hashtag_id of this class : 2107755900 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1334557442, 1334557361] Looping around the photos to save general results len do output : 116 /1334562812Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562813Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562814Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562815Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562816Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562817Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562818Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562819Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562820Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562821Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562822Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562823Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562825Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562826Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562827Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562828Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562829Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562830Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562831Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562832Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562834Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562835Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562836Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562837Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562838Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562839Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562840Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562842Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562843Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562844Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562845Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562846Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562847Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562848Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562849Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562850Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562851Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562853Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562854Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562855Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562856Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562857Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562858Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562859Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562860Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562861Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562862Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562863Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562870Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562871Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562872Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562873Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562874Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562875Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562876Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562877Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562878Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562879Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562880Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562881Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562882Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562883Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562884Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562885Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562886Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562887Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562888Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562889Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562890Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562891Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562892Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562893Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562894Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562895Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562896Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562897Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562901Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562903Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562904Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562905Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562906Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562907Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562908Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562909Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562910Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562911Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562912Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562913Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562956Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562960Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562964Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562968Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562972Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562976Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562980Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562984Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562988Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562991Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562995Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562997Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562998Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334562999Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334563000Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334563001Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334563002Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334563004Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334563005Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334563006Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334563007Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334563008Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334563010Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334563011Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334563012Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334563013Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334563014Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334563015Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2545063') ('4234', '20210830', '1334557442', None, None, None, None, None, '2545063') ('4234', None, None, None, None, None, None, None, '2545063') ('4234', '20210830', '1334557361', None, None, None, None, None, '2545063') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 350 time used for this insertion : 0.029901742935180664 save_final save missing photos in datou_result : time spend for datou_step_exec : 29.419915437698364 time spend to save output : 0.03441214561462402 total time spend for step 5 : 29.45432758331299 step6:thcl Tue Feb 4 14:21:32 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! we are using the classfication for only one thcl 3237 time to import caffe and check if the image exist : 0.010092020034790039 time to convert the images to numpy array : 0.0055315494537353516 time to import caffe and check if the image exist : 0.016153335571289062 time to convert the images to numpy array : 0.003855466842651367 time to import caffe and check if the image exist : 0.0020987987518310547 time to convert the images to numpy array : 0.02716541290283203 time to import caffe and check if the image exist : 0.020734786987304688 time to convert the images to numpy array : 0.013891935348510742 time to import caffe and check if the image exist : 0.0086517333984375 time to convert the images to numpy array : 0.032796621322631836 time to import caffe and check if the image exist : 0.014260530471801758 time to convert the images to numpy array : 0.03102898597717285 time to import caffe and check if the image exist : 0.0065081119537353516 time to convert the images to numpy array : 0.041840314865112305 time to import caffe and check if the image exist : 0.031165122985839844 time to convert the images to numpy array : 0.021692514419555664 time to import caffe and check if the image exist : 0.02780747413635254 time to convert the images to numpy array : 0.025990724563598633 time to import caffe and check if the image exist : 0.028443574905395508 time to convert the images to numpy array : 0.025852203369140625 total time to convert the images to numpy array : 0.4024810791015625 list photo_ids error: [] list photo_ids correct : [1334562910, 1334562911, 1334562912, 1334562913, 1334562956, 1334562960, 1334562964, 1334562968, 1334562972, 1334562976, 1334562980, 1334562984, 1334562882, 1334562883, 1334562884, 1334562885, 1334562886, 1334562887, 1334562888, 1334562889, 1334562890, 1334562891, 1334562892, 1334562893, 1334562894, 1334562895, 1334562896, 1334562897, 1334562901, 1334562903, 1334562904, 1334562905, 1334562906, 1334562907, 1334562908, 1334562909, 1334562825, 1334562826, 1334562827, 1334562828, 1334562829, 1334562830, 1334562831, 1334562832, 1334562834, 1334562835, 1334562836, 1334562837, 1334563007, 1334563008, 1334563010, 1334563011, 1334563012, 1334563013, 1334563014, 1334563015, 1334562988, 1334562991, 1334562995, 1334562997, 1334562998, 1334562999, 1334563000, 1334563001, 1334563002, 1334563004, 1334563005, 1334563006, 1334562812, 1334562813, 1334562814, 1334562815, 1334562816, 1334562817, 1334562818, 1334562819, 1334562820, 1334562821, 1334562822, 1334562823, 1334562838, 1334562839, 1334562840, 1334562842, 1334562843, 1334562844, 1334562845, 1334562846, 1334562847, 1334562848, 1334562849, 1334562850, 1334562870, 1334562871, 1334562872, 1334562873, 1334562874, 1334562875, 1334562876, 1334562877, 1334562878, 1334562879, 1334562880, 1334562881, 1334562851, 1334562853, 1334562854, 1334562855, 1334562856, 1334562857, 1334562858, 1334562859, 1334562860, 1334562861, 1334562862, 1334562863] number of photos to traite : 116 try to delete the photos incorrect in DB tagging for thcl : 3237 To do loadFromThcl(), then load ParamDescType : thcl3237 thcls : [{'id': 3237, 'mtr_user_id': 31, 'name': 'learn_rubbia_refus_2500', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton,Film_plastique,PEHD,PET_clair,PET_fonce,Papier,Tetrapak,flou,mal_croppe,metal,refus', 'svm_portfolios_learning': '4865689,4865690,4865686,4865684,4865685,4865688,4865691,4865693,4865692,4865687,4865683', 'photo_hashtag_type': 4158, 'photo_desc_type': 5561, 'type_classification': 'caffe', 'hashtag_id_list': '492774966,2107756122,628944319,2107755846,2107755900,492668766,609991870,492777938,2107755527,492628673,538914404'}] thcl {'id': 3237, 'mtr_user_id': 31, 'name': 'learn_rubbia_refus_2500', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton,Film_plastique,PEHD,PET_clair,PET_fonce,Papier,Tetrapak,flou,mal_croppe,metal,refus', 'svm_portfolios_learning': '4865689,4865690,4865686,4865684,4865685,4865688,4865691,4865693,4865692,4865687,4865683', 'photo_hashtag_type': 4158, 'photo_desc_type': 5561, 'type_classification': 'caffe', 'hashtag_id_list': '492774966,2107756122,628944319,2107755846,2107755900,492668766,609991870,492777938,2107755527,492628673,538914404'} Update svm_hashtag_type_desc : 5561 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5561, 'learn_rubbia_refus_2500', 2048, 2048, 'learn_rubbia_refus_2500', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2021, 12, 2, 19, 10, 8), datetime.datetime(2021, 12, 2, 19, 10, 8)) To loadFromThcl() : net_5561 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10553 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5561, 'learn_rubbia_refus_2500', 2048, 2048, 'learn_rubbia_refus_2500', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2021, 12, 2, 19, 10, 8), datetime.datetime(2021, 12, 2, 19, 10, 8)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_rubbia_refus_2500 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_rubbia_refus_2500 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_rubbia_refus_2500 /data/models_weight/learn_rubbia_refus_2500/caffemodel size_local : 94358479 size in s3 : 94358479 create time local : 2021-12-03 18:29:39 create time in s3 : 2021-12-02 17:49:16 caffemodel already exist and didn't need to update /data/models_weight/learn_rubbia_refus_2500/deploy.prototxt size_local : 32544 size in s3 : 32544 create time local : 2021-12-03 18:29:39 create time in s3 : 2021-12-02 17:49:15 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_rubbia_refus_2500/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-12-03 18:29:40 create time in s3 : 2021-12-02 18:09:52 mean.npy already exist and didn't need to update /data/models_weight/learn_rubbia_refus_2500/synset_words.txt size_local : 334 size in s3 : 334 create time local : 2021-12-03 18:29:40 create time in s3 : 2021-12-02 18:10:06 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_rubbia_refus_2500/deploy.prototxt caffemodel_filename : /data/models_weight/learn_rubbia_refus_2500/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10334 max_wait_temp : 1 max_wait : 0 dict_keys(['prob', 'pool5']) time used to do the prepocess of the images : 0.6248962879180908 time used to do the prediction : 0.5432283878326416 save descriptor for thcl : 3237 time to traite the descriptors : 0.7150721549987793 Catched exception ! Connect or reconnect ! storage_type for insertDescriptorsMulti : 3 To insert : 1334562910 To insert : 1334562911 To insert : 1334562912 To insert : 1334562913 To insert : 1334562956 To insert : 1334562960 To insert : 1334562964 To insert : 1334562968 To insert : 1334562972 To insert : 1334562976 To insert : 1334562980 To insert : 1334562984 To insert : 1334562882 To insert : 1334562883 To insert : 1334562884 To insert : 1334562885 To insert : 1334562886 To insert : 1334562887 To insert : 1334562888 To insert : 1334562889 To insert : 1334562890 To insert : 1334562891 To insert : 1334562892 To insert : 1334562893 To insert : 1334562894 To insert : 1334562895 To insert : 1334562896 To insert : 1334562897 To insert : 1334562901 To insert : 1334562903 To insert : 1334562904 To insert : 1334562905 To insert : 1334562906 To insert : 1334562907 To insert : 1334562908 To insert : 1334562909 To insert : 1334562825 To insert : 1334562826 To insert : 1334562827 To insert : 1334562828 To insert : 1334562829 To insert : 1334562830 To insert : 1334562831 To insert : 1334562832 To insert : 1334562834 To insert : 1334562835 To insert : 1334562836 To insert : 1334562837 To insert : 1334563007 To insert : 1334563008 To insert : 1334563010 To insert : 1334563011 To insert : 1334563012 To insert : 1334563013 To insert : 1334563014 To insert : 1334563015 To insert : 1334562988 To insert : 1334562991 To insert : 1334562995 To insert : 1334562997 To insert : 1334562998 To insert : 1334562999 To insert : 1334563000 To insert : 1334563001 To insert : 1334563002 To insert : 1334563004 To insert : 1334563005 To insert : 1334563006 To insert : 1334562812 To insert : 1334562813 To insert : 1334562814 To insert : 1334562815 To insert : 1334562816 To insert : 1334562817 To insert : 1334562818 To insert : 1334562819 To insert : 1334562820 To insert : 1334562821 To insert : 1334562822 To insert : 1334562823 To insert : 1334562838 To insert : 1334562839 To insert : 1334562840 To insert : 1334562842 To insert : 1334562843 To insert : 1334562844 To insert : 1334562845 To insert : 1334562846 To insert : 1334562847 To insert : 1334562848 To insert : 1334562849 To insert : 1334562850 To insert : 1334562870 To insert : 1334562871 To insert : 1334562872 To insert : 1334562873 To insert : 1334562874 To insert : 1334562875 To insert : 1334562876 To insert : 1334562877 To insert : 1334562878 To insert : 1334562879 To insert : 1334562880 To insert : 1334562881 To insert : 1334562851 To insert : 1334562853 To insert : 1334562854 To insert : 1334562855 To insert : 1334562856 To insert : 1334562857 To insert : 1334562858 To insert : 1334562859 To insert : 1334562860 To insert : 1334562861 To insert : 1334562862 To insert : 1334562863 time to insert the descriptors : 24.187870979309082 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos SAVE THCL : begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 182 time used for this insertion : 0.018939495086669922 save missing photos in datou_result : time spend for datou_step_exec : 30.587758541107178 time spend to save output : 0.06462645530700684 total time spend for step 6 : 30.652384996414185 step7:ventilate_hashtags_in_portfolio Tue Feb 4 14:22:03 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! Iterating over portfolio : 20210830 get user id for portfolio 20210830 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20210830 AND mptpi.`type`=4230 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','mal_croppe','Film_plastique','PEHD','Carton','pet_clair','PET_clair','PET_fonce','metal','Papier','environnement','autre','flou','pet_fonce','pehd','Tetrapak','carton','refus')) AND mptpi.`min_score`=0.6 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20210830 AND mptpi.`type`=4230 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','mal_croppe','Film_plastique','PEHD','Carton','pet_clair','PET_clair','PET_fonce','metal','Papier','environnement','autre','flou','pet_fonce','pehd','Tetrapak','carton','refus')) AND mptpi.`min_score`=0.6 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20210830 AND mptpi.`type`=4231 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','mal_croppe','Film_plastique','PEHD','Carton','pet_clair','PET_clair','PET_fonce','metal','Papier','environnement','autre','flou','pet_fonce','pehd','Tetrapak','carton','refus')) AND mptpi.`min_score`=0.6 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20210830 AND mptpi.`type`=4231 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','mal_croppe','Film_plastique','PEHD','Carton','pet_clair','PET_clair','PET_fonce','metal','Papier','environnement','autre','flou','pet_fonce','pehd','Tetrapak','carton','refus')) AND mptpi.`min_score`=0.6 To do lien utilise dans velours : https://www.fotonower.com/velours/20211205,20211197,20211198,20211210,20211212,20211202,20211209,20211204,20211206,20211207,20211208,20211211,20211213?tags=papier,mal_croppe,film_plastique,pehd,carton,pet_clair,pet_fonce,metal,environnement,autre,flou,tetrapak,refus&datou_id_consolidate=4235&port_consolidate=20210830 Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1334557442, 1334557361] Looping around the photos to save general results len do output : 1 /20210830. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2545063') ('4234', '20210830', '1334557442', None, None, None, None, None, '2545063') ('4234', None, None, None, None, None, None, None, '2545063') ('4234', '20210830', '1334557361', None, None, None, None, None, '2545063') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 time used for this insertion : 0.012614727020263672 save_final save missing photos in datou_result : time spend for datou_step_exec : 2.399186611175537 time spend to save output : 0.012885332107543945 total time spend for step 7 : 2.412071943283081 step8:final Tue Feb 4 14:22:05 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1334557442: ('0.22742143331369938',), 1334557361: ('0.22742143331369938',)} new output for save of step final : {1334557442: ('0.22742143331369938',), 1334557361: ('0.22742143331369938',)} [1334557442, 1334557361] Looping around the photos to save general results len do output : 2 /1334557442.Didn't retrieve data . /1334557361.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2545063') ('4234', '20210830', '1334557442', None, None, None, None, None, '2545063') ('4234', None, None, None, None, None, None, None, '2545063') ('4234', '20210830', '1334557361', None, None, None, None, None, '2545063') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 6 time used for this insertion : 0.012760639190673828 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.11111760139465332 time spend to save output : 0.01296854019165039 total time spend for step 8 : 0.12408614158630371 step9:velours_tree Tue Feb 4 14:22:06 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure list_portfolios : 20211205,20211197,20211198,20211210,20211212,20211202,20211209,20211204,20211206,20211207,20211208,20211211,20211213 photo desc type : 5561 - Retrieving photos to tag... query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20211205 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20211197 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20211198 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20211210 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20211212 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20211202 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20211209 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20211204 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20211206 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20211207 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20211208 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20211211 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20211213 ORDER BY ph.size desc - Loading descriptors... Size : 2048 len(descriptors) : 116 Compute structured hierarchical clustering... ward : AgglomerativeClustering(n_clusters=6) ward.labels_ : [5 5 1 1 5 0 2 2 1 2 2 2 4 4 2 4 1 4 0 2 4 4 4 3 3 4 5 1 0 3 2 1 2 4 2 4 0 4 0 2 4 4 0 4 3 2 3 3 0 3 2 2 3 0 2 3 3 4 3 4 3 0 2 2 2 3 4 3 3 0 2 3 3 3 3 3 2 1 2 0 4 2 0 1 0 2 4 4 4 5 4 5 3 4 4 0 3 1 4 3 4 4 4 4 5 5 4 5 3 3 1 4 4 4 4 4] Elapsed time: 0.022942304611206055 graph_id used : 75982 - Beta version, working pretty good on 11-5-16 ! https://www.fotonower.com/velours/20211205,20211197,20211198,20211210,20211212,20211202,20211209,20211204,20211206,20211207,20211208,20211211,20211213?tags=papier,mal_croppe,film_plastique,pehd,carton,pet_clair,pet_fonce,metal,environnement,autre,flou,tetrapak,refus&datou_id_consolidate=4235&port_consolidate=20210830&tree_id=75982 Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : velours_tree we use saveGeneral [1334557442, 1334557361] Looping around the photos to save general results len do output : 1 /20210830Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2545063') ('4234', '20210830', '1334557442', None, None, None, None, None, '2545063') ('4234', None, None, None, None, None, None, None, '2545063') ('4234', '20210830', '1334557361', None, None, None, None, None, '2545063') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 time used for this insertion : 0.012102127075195312 save_final save missing photos in datou_result : time spend for datou_step_exec : 7.3339197635650635 time spend to save output : 0.012500762939453125 total time spend for step 9 : 7.346420526504517 step10:send_mail_cod Tue Feb 4 14:22:13 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Qualipapia_P20210830_04-02-2025_14_22_13.pdf 20211187 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202111871738675333 20211179 imagette202111791738675334 20211180 imagette202111801738675334 20211192 imagette202111921738675334 20211194 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202111941738675334 20211184 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202111841738675335 20211191 imagette202111911738675335 20211186 imagette202111861738675335 20211189 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202111891738675335 20211190 imagette202111901738675336 20211193 imagette202111931738675336 20211195 imagette202111951738675336 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20210830 and hashtag_type = 4230 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20211205,20211197,20211198,20211210,20211212,20211202,20211209,20211204,20211206,20211207,20211208,20211211,20211213?tags=papier,mal_croppe,film_plastique,pehd,carton,pet_clair,pet_fonce,metal,environnement,autre,flou,tetrapak,refus&datou_id_consolidate=4235&port_consolidate=20210830&tree_id=75982 your option no_mail is active, we will not send the real mail to your client args[1334557442] : ((1334557442, -7.720226752089334, 492609224), (1334557442, -0.5066478628093157, 501862349), '0.22742143331369938') We are sending mail with results at cod@fotonower.com args[1334557361] : ((1334557361, -7.688103685602205, 492609224), (1334557361, -0.5244189784598533, 501862349), '0.22742143331369938') We are sending mail with results at cod@fotonower.com refus_total : 0.22742143331369938 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20210830 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20210830_04-02-2025_14_22_13.pdf results_Qualipapia_P20210830_04-02-2025_14_22_13.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20210830_04-02-2025_14_22_13.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('4234','20210830','results_Qualipapia_P20210830_04-02-2025_14_22_13.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20210830_04-02-2025_14_22_13.pdf','pdf','','0.15','0.22742143331369938') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1334557442, 1334557361] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2545063') ('4234', '20210830', '1334557442', None, None, None, None, None, '2545063') ('4234', None, None, None, None, None, None, None, '2545063') ('4234', '20210830', '1334557361', None, None, None, None, None, '2545063') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 2 time used for this insertion : 0.011249303817749023 save_final save missing photos in datou_result : time spend for datou_step_exec : 4.100013971328735 time spend to save output : 0.01149749755859375 total time spend for step 10 : 4.111511468887329 step11:split_time_score Tue Feb 4 14:22:17 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 3442, 'mtr_user_id': 31, 'name': 'classifieur_2camions_valcor_021122_v1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'deux_camions,camion_droite,camion_gauche,pas_de_camion', 'svm_portfolios_learning': '7659379,7659034,7657685,7657114', 'photo_hashtag_type': 4458, 'photo_desc_type': 5723, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107760533,2107760534,2107760535,2107760536'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('11', 2),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 04022025 20210830 Nombre de photos uploadées : 2 / 23040 (0%) 04022025 20210830 Nombre de photos taguées (types de déchets): 2 / 2 (100%) 04022025 20210830 Nombre de photos taguées (volume) : 0 / 2 (0%) elapsed_time : load_data_split_time_score 3.0994415283203125e-06 elapsed_time : order_list_meta_photo_and_scores 7.62939453125e-06 ?? elapsed_time : fill_and_build_computed_from_old_data 0.00028634071350097656 elapsed_time : insert_dashboard_record_day_entry 0.022577285766601562 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20209107 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20209605 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20209607 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20209609 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20209611 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20209613 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20209615 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20209634 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20209653 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20209654 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20209655 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20209656 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20209657 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20209659 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20209664 order by id desc limit 1 Qualite : 0.2069396565880867 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20209677_04-02-2025_13_17_29.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20209677 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20209677 AND mptpi.`type`=4230 To do Qualite : 0.20080662994765655 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20209678_04-02-2025_13_24_18.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20209678 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20209678 AND mptpi.`type`=4230 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210180 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210182 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210184 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210187 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210188 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210189 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210191 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210193 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210195 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210197 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210198 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210199 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210200 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210201 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210202 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210203 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210204 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210205 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210206 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210207 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210208 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210209 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210210 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210211 order by id desc limit 1 Qualite : 0.20048873371300646 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20210212_04-02-2025_14_02_56.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210212 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20210212 AND mptpi.`type`=4230 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210817 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210819 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210821 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210823 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210825 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210827 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210829 order by id desc limit 1 Qualite : 0.22742143331369938 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20210830_04-02-2025_14_22_13.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20210830 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20210830 AND mptpi.`type`=4230 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'04022025': {'nb_upload': 2, 'nb_taggue_class': 2, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1334557442, 1334557361] Looping around the photos to save general results len do output : 1 /20210830Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2545063') ('4234', '20210830', '1334557442', None, None, None, None, None, '2545063') ('4234', None, None, None, None, None, None, None, '2545063') ('4234', '20210830', '1334557361', None, None, None, None, None, '2545063') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 time used for this insertion : 0.012381315231323242 save_final save missing photos in datou_result : time spend for datou_step_exec : 36.98179888725281 time spend to save output : 0.01252603530883789 total time spend for step 11 : 36.994324922561646 caffe_path_current : About to save ! 2 After save, about to update current ! ret : 1 len(input) + len(total_photo_id_missing) : 2 set_done_treatment 33.27user 21.09system 2:28.86elapsed 36%CPU (0avgtext+0avgdata 2495224maxresident)k 927128inputs+12048outputs (1126major+1996779minor)pagefaults 0swaps