python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 4234 -l 5 ' -s datou_current_4234 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2987498 load datou : 4234 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5275, 'learn_RUBBIA_REFUS_AMIENS_23', 16384, 25088, 'learn_RUBBIA_REFUS_AMIENS_23', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 23, 14, 19, 39), datetime.datetime(2021, 4, 23, 14, 19, 39)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5561, 'learn_rubbia_refus_2500', 2048, 2048, 'learn_rubbia_refus_2500', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2021, 12, 2, 19, 10, 8), datetime.datetime(2021, 12, 2, 19, 10, 8)) load thcls load THCL from format json or kwargs add thcl : 2847 in CacheModelConfig load THCL from format json or kwargs add thcl : 3237 in CacheModelConfig load pdts add pdt : 5275 in CacheModelConfig add pdt : 5561 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 list_input_json: [] Current got : datou_id : 4234, datou_cur_ids : ['2576510'] with mtr_portfolio_ids : ['20447187'] and first list_photo_ids : [] new path : /proc/2987498/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, brightness, blur_detection, rle_unique_nms_with_priority, crop_condition, thcl, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , BFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 2 ; length of list_pids : 2 ; length of list_args : 2 time to download the photos : 0.2547414302825928 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Tue Feb 11 15:20:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10553 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-11 15:20:32.780269: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-11 15:20:32.807174: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-11 15:20:32.809309: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fb3e8000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-11 15:20:32.809337: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-11 15:20:32.813388: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-11 15:20:33.057525: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x26075160 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-11 15:20:33.057592: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-11 15:20:33.058890: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-11 15:20:33.059350: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-11 15:20:33.062089: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-11 15:20:33.064912: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-11 15:20:33.065446: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-11 15:20:33.068645: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-11 15:20:33.070160: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-11 15:20:33.076606: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-11 15:20:33.078653: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-11 15:20:33.078803: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-11 15:20:33.079882: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-11 15:20:33.079908: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-11 15:20:33.079921: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-11 15:20:33.081718: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9777 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-11 15:20:33.370970: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-11 15:20:33.371127: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-11 15:20:33.371151: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-11 15:20:33.371172: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-11 15:20:33.371191: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-11 15:20:33.371210: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-11 15:20:33.371229: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-11 15:20:33.371248: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-11 15:20:33.372863: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-11 15:20:33.374389: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-11 15:20:33.374457: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-11 15:20:33.374480: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-11 15:20:33.374501: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-11 15:20:33.374521: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-11 15:20:33.374541: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-11 15:20:33.374561: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-11 15:20:33.374582: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-11 15:20:33.376188: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-11 15:20:33.376229: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-11 15:20:33.376240: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-11 15:20:33.376250: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-11 15:20:33.377908: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9777 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2847 thcls : [{'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}] thcl {'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5275 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5275, 'learn_RUBBIA_REFUS_AMIENS_23', 16384, 25088, 'learn_RUBBIA_REFUS_AMIENS_23', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 23, 14, 19, 39), datetime.datetime(2021, 4, 23, 14, 19, 39)) {'thcl': {'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'], 'list_hashtags_csv': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'svm_hashtag_type_desc': 5275, 'photo_desc_type': 5275, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_RUBBIA_REFUS_AMIENS_23 NUM_CLASSES 9 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_RUBBIA_REFUS_AMIENS_23 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-11 15:20:40.192261: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-11 15:20:40.362331: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_RUBBIA_REFUS_AMIENS_23 /data/models_weight/learn_RUBBIA_REFUS_AMIENS_23/mask_model.h5 size_local : 256009536 size in s3 : 256009536 create time local : 2021-08-09 09:43:22 create time in s3 : 2021-08-06 18:54:04 mask_model.h5 already exist and didn't need to update list_images length : 2 NEW PHOTO pour l'instant on ne peut pas sauvegarder la photo dans les tile Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -118.55547 max: 148.37344 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 8 time for calcul the mask position with numpy : 7.772445678710938e-05 nb_pixel_total : 922 time to create 1 rle with old method : 0.001760721206665039 length of segment : 31 time for calcul the mask position with numpy : 5.0067901611328125e-05 nb_pixel_total : 449 time to create 1 rle with old method : 0.0008575916290283203 length of segment : 21 time for calcul the mask position with numpy : 4.553794860839844e-05 nb_pixel_total : 73 time to create 1 rle with old method : 0.00015687942504882812 length of segment : 22 time for calcul the mask position with numpy : 0.00037789344787597656 nb_pixel_total : 19777 time to create 1 rle with old method : 0.03196454048156738 length of segment : 177 time for calcul the mask position with numpy : 3.6716461181640625e-05 nb_pixel_total : 145 time to create 1 rle with old method : 0.00019431114196777344 length of segment : 33 time for calcul the mask position with numpy : 4.4345855712890625e-05 nb_pixel_total : 833 time to create 1 rle with old method : 0.0010187625885009766 length of segment : 33 time for calcul the mask position with numpy : 2.9802322387695312e-05 nb_pixel_total : 134 time to create 1 rle with old method : 0.00017905235290527344 length of segment : 41 time for calcul the mask position with numpy : 2.6464462280273438e-05 nb_pixel_total : 11 time to create 1 rle with old method : 6.604194641113281e-05 length of segment : 3 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -120.01641 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 7 time for calcul the mask position with numpy : 7.963180541992188e-05 nb_pixel_total : 559 time to create 1 rle with old method : 0.0011932849884033203 length of segment : 51 time for calcul the mask position with numpy : 6.246566772460938e-05 nb_pixel_total : 2317 time to create 1 rle with old method : 0.00299072265625 length of segment : 36 time for calcul the mask position with numpy : 0.00012755393981933594 nb_pixel_total : 5384 time to create 1 rle with old method : 0.006842851638793945 length of segment : 70 time for calcul the mask position with numpy : 7.343292236328125e-05 nb_pixel_total : 1571 time to create 1 rle with old method : 0.0020885467529296875 length of segment : 68 time for calcul the mask position with numpy : 8.344650268554688e-05 nb_pixel_total : 2138 time to create 1 rle with old method : 0.002702474594116211 length of segment : 63 time for calcul the mask position with numpy : 4.38690185546875e-05 nb_pixel_total : 784 time to create 1 rle with old method : 0.001005411148071289 length of segment : 36 time for calcul the mask position with numpy : 6.818771362304688e-05 nb_pixel_total : 2402 time to create 1 rle with old method : 0.003154754638671875 length of segment : 35 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -111.25859 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 4 time for calcul the mask position with numpy : 0.00010514259338378906 nb_pixel_total : 6260 time to create 1 rle with old method : 0.007439851760864258 length of segment : 120 time for calcul the mask position with numpy : 6.961822509765625e-05 nb_pixel_total : 2501 time to create 1 rle with old method : 0.003386974334716797 length of segment : 42 time for calcul the mask position with numpy : 4.029273986816406e-05 nb_pixel_total : 705 time to create 1 rle with old method : 0.000978231430053711 length of segment : 29 time for calcul the mask position with numpy : 8.273124694824219e-05 nb_pixel_total : 3548 time to create 1 rle with old method : 0.0046558380126953125 length of segment : 50 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -122.14141 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 3 time for calcul the mask position with numpy : 4.6253204345703125e-05 nb_pixel_total : 504 time to create 1 rle with old method : 0.0008065700531005859 length of segment : 28 time for calcul the mask position with numpy : 3.981590270996094e-05 nb_pixel_total : 318 time to create 1 rle with old method : 0.0004851818084716797 length of segment : 39 time for calcul the mask position with numpy : 0.00015997886657714844 nb_pixel_total : 10923 time to create 1 rle with old method : 0.012970685958862305 length of segment : 87 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 220.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 99.15859 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 3 time for calcul the mask position with numpy : 4.4345855712890625e-05 nb_pixel_total : 518 time to create 1 rle with old method : 0.0007207393646240234 length of segment : 20 time for calcul the mask position with numpy : 3.147125244140625e-05 nb_pixel_total : 94 time to create 1 rle with old method : 0.00017905235290527344 length of segment : 10 time for calcul the mask position with numpy : 0.0006940364837646484 nb_pixel_total : 68420 time to create 1 rle with old method : 0.07703137397766113 length of segment : 299 Processing 1 images image shape: (400, 400, 3) min: 14.00000 max: 208.00000 molded_images shape: (1, 640, 640, 3) min: -83.32891 max: 75.39531 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 0 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -106.66875 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 10 time for calcul the mask position with numpy : 6.008148193359375e-05 nb_pixel_total : 1824 time to create 1 rle with old method : 0.0023348331451416016 length of segment : 30 time for calcul the mask position with numpy : 4.076957702636719e-05 nb_pixel_total : 406 time to create 1 rle with old method : 0.0006253719329833984 length of segment : 15 time for calcul the mask position with numpy : 4.5299530029296875e-05 nb_pixel_total : 419 time to create 1 rle with old method : 0.0007424354553222656 length of segment : 22 time for calcul the mask position with numpy : 5.14984130859375e-05 nb_pixel_total : 844 time to create 1 rle with old method : 0.0011105537414550781 length of segment : 31 time for calcul the mask position with numpy : 7.104873657226562e-05 nb_pixel_total : 2725 time to create 1 rle with old method : 0.0035414695739746094 length of segment : 48 time for calcul the mask position with numpy : 3.719329833984375e-05 nb_pixel_total : 396 time to create 1 rle with old method : 0.0006089210510253906 length of segment : 18 time for calcul the mask position with numpy : 3.123283386230469e-05 nb_pixel_total : 133 time to create 1 rle with old method : 0.00022983551025390625 length of segment : 11 time for calcul the mask position with numpy : 3.314018249511719e-05 nb_pixel_total : 232 time to create 1 rle with old method : 0.0004029273986816406 length of segment : 18 time for calcul the mask position with numpy : 4.029273986816406e-05 nb_pixel_total : 840 time to create 1 rle with old method : 0.0010533332824707031 length of segment : 32 time for calcul the mask position with numpy : 4.7206878662109375e-05 nb_pixel_total : 697 time to create 1 rle with old method : 0.0009355545043945312 length of segment : 39 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -117.38750 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 4 time for calcul the mask position with numpy : 5.1021575927734375e-05 nb_pixel_total : 738 time to create 1 rle with old method : 0.0010499954223632812 length of segment : 30 time for calcul the mask position with numpy : 4.553794860839844e-05 nb_pixel_total : 1051 time to create 1 rle with old method : 0.0014028549194335938 length of segment : 69 time for calcul the mask position with numpy : 0.0001881122589111328 nb_pixel_total : 9926 time to create 1 rle with old method : 0.011787176132202148 length of segment : 167 time for calcul the mask position with numpy : 3.528594970703125e-05 nb_pixel_total : 164 time to create 1 rle with old method : 0.0002696514129638672 length of segment : 15 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -114.80547 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 7 time for calcul the mask position with numpy : 8.988380432128906e-05 nb_pixel_total : 2141 time to create 1 rle with old method : 0.0026030540466308594 length of segment : 135 time for calcul the mask position with numpy : 6.341934204101562e-05 nb_pixel_total : 1036 time to create 1 rle with old method : 0.001428365707397461 length of segment : 40 time for calcul the mask position with numpy : 4.553794860839844e-05 nb_pixel_total : 1168 time to create 1 rle with old method : 0.0014505386352539062 length of segment : 37 time for calcul the mask position with numpy : 0.00032806396484375 nb_pixel_total : 16878 time to create 1 rle with old method : 0.019780635833740234 length of segment : 269 time for calcul the mask position with numpy : 6.699562072753906e-05 nb_pixel_total : 29 time to create 1 rle with old method : 9.799003601074219e-05 length of segment : 14 time for calcul the mask position with numpy : 3.9577484130859375e-05 nb_pixel_total : 852 time to create 1 rle with old method : 0.0010690689086914062 length of segment : 37 time for calcul the mask position with numpy : 7.653236389160156e-05 nb_pixel_total : 1991 time to create 1 rle with old method : 0.002238750457763672 length of segment : 122 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 246.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 124.92266 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 9 time for calcul the mask position with numpy : 4.673004150390625e-05 nb_pixel_total : 918 time to create 1 rle with old method : 0.0011315345764160156 length of segment : 49 time for calcul the mask position with numpy : 8.535385131835938e-05 nb_pixel_total : 2411 time to create 1 rle with old method : 0.003019571304321289 length of segment : 148 time for calcul the mask position with numpy : 4.410743713378906e-05 nb_pixel_total : 903 time to create 1 rle with old method : 0.0011317729949951172 length of segment : 48 time for calcul the mask position with numpy : 3.457069396972656e-05 nb_pixel_total : 266 time to create 1 rle with old method : 0.0003933906555175781 length of segment : 18 time for calcul the mask position with numpy : 6.794929504394531e-05 nb_pixel_total : 666 time to create 1 rle with old method : 0.0009374618530273438 length of segment : 81 time for calcul the mask position with numpy : 7.367134094238281e-05 nb_pixel_total : 1865 time to create 1 rle with old method : 0.0023763179779052734 length of segment : 152 time for calcul the mask position with numpy : 3.24249267578125e-05 nb_pixel_total : 109 time to create 1 rle with old method : 0.00020742416381835938 length of segment : 15 time for calcul the mask position with numpy : 5.459785461425781e-05 nb_pixel_total : 1339 time to create 1 rle with old method : 0.0016434192657470703 length of segment : 75 time for calcul the mask position with numpy : 0.00015878677368164062 nb_pixel_total : 3991 time to create 1 rle with old method : 0.004963874816894531 length of segment : 229 Processing 1 images image shape: (280, 400, 3) min: 11.00000 max: 200.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 65.59844 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 1 time for calcul the mask position with numpy : 0.0011758804321289062 nb_pixel_total : 105767 time to create 1 rle with old method : 0.12642431259155273 length of segment : 281 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 149.51406 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 19 time for calcul the mask position with numpy : 9.608268737792969e-05 nb_pixel_total : 457 time to create 1 rle with old method : 0.0008747577667236328 length of segment : 27 time for calcul the mask position with numpy : 5.507469177246094e-05 nb_pixel_total : 302 time to create 1 rle with old method : 0.0006241798400878906 length of segment : 39 time for calcul the mask position with numpy : 0.00013947486877441406 nb_pixel_total : 2428 time to create 1 rle with old method : 0.004351377487182617 length of segment : 68 time for calcul the mask position with numpy : 8.893013000488281e-05 nb_pixel_total : 539 time to create 1 rle with old method : 0.0013473033905029297 length of segment : 38 time for calcul the mask position with numpy : 0.00010275840759277344 nb_pixel_total : 3369 time to create 1 rle with old method : 0.004384756088256836 length of segment : 62 time for calcul the mask position with numpy : 4.1961669921875e-05 nb_pixel_total : 430 time to create 1 rle with old method : 0.0005943775177001953 length of segment : 27 time for calcul the mask position with numpy : 3.552436828613281e-05 nb_pixel_total : 241 time to create 1 rle with old method : 0.0003478527069091797 length of segment : 16 time for calcul the mask position with numpy : 4.482269287109375e-05 nb_pixel_total : 717 time to create 1 rle with old method : 0.0008997917175292969 length of segment : 34 time for calcul the mask position with numpy : 3.504753112792969e-05 nb_pixel_total : 122 time to create 1 rle with old method : 0.00022268295288085938 length of segment : 15 time for calcul the mask position with numpy : 3.9577484130859375e-05 nb_pixel_total : 538 time to create 1 rle with old method : 0.0006945133209228516 length of segment : 29 time for calcul the mask position with numpy : 5.817413330078125e-05 nb_pixel_total : 1143 time to create 1 rle with old method : 0.0015149116516113281 length of segment : 33 time for calcul the mask position with numpy : 5.221366882324219e-05 nb_pixel_total : 1172 time to create 1 rle with old method : 0.0015499591827392578 length of segment : 49 time for calcul the mask position with numpy : 3.647804260253906e-05 nb_pixel_total : 365 time to create 1 rle with old method : 0.0004754066467285156 length of segment : 27 time for calcul the mask position with numpy : 3.695487976074219e-05 nb_pixel_total : 371 time to create 1 rle with old method : 0.0005095005035400391 length of segment : 18 time for calcul the mask position with numpy : 3.457069396972656e-05 nb_pixel_total : 274 time to create 1 rle with old method : 0.00033664703369140625 length of segment : 25 time for calcul the mask position with numpy : 4.76837158203125e-05 nb_pixel_total : 79 time to create 1 rle with old method : 0.00013017654418945312 length of segment : 14 time for calcul the mask position with numpy : 4.839897155761719e-05 nb_pixel_total : 1111 time to create 1 rle with old method : 0.0013456344604492188 length of segment : 48 time for calcul the mask position with numpy : 7.081031799316406e-05 nb_pixel_total : 3162 time to create 1 rle with old method : 0.00363922119140625 length of segment : 59 time for calcul the mask position with numpy : 3.7670135498046875e-05 nb_pixel_total : 347 time to create 1 rle with old method : 0.0004246234893798828 length of segment : 25 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 150.73672 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 14 time for calcul the mask position with numpy : 4.792213439941406e-05 nb_pixel_total : 664 time to create 1 rle with old method : 0.0008220672607421875 length of segment : 41 time for calcul the mask position with numpy : 3.695487976074219e-05 nb_pixel_total : 425 time to create 1 rle with old method : 0.0006318092346191406 length of segment : 19 time for calcul the mask position with numpy : 3.4809112548828125e-05 nb_pixel_total : 389 time to create 1 rle with old method : 0.0005252361297607422 length of segment : 22 time for calcul the mask position with numpy : 4.363059997558594e-05 nb_pixel_total : 761 time to create 1 rle with old method : 0.0009462833404541016 length of segment : 56 time for calcul the mask position with numpy : 5.030632019042969e-05 nb_pixel_total : 1382 time to create 1 rle with old method : 0.001809835433959961 length of segment : 43 time for calcul the mask position with numpy : 3.600120544433594e-05 nb_pixel_total : 417 time to create 1 rle with old method : 0.0006008148193359375 length of segment : 20 time for calcul the mask position with numpy : 3.337860107421875e-05 nb_pixel_total : 257 time to create 1 rle with old method : 0.0003681182861328125 length of segment : 19 time for calcul the mask position with numpy : 3.361701965332031e-05 nb_pixel_total : 230 time to create 1 rle with old method : 0.00039386749267578125 length of segment : 22 time for calcul the mask position with numpy : 3.0994415283203125e-05 nb_pixel_total : 140 time to create 1 rle with old method : 0.00022649765014648438 length of segment : 13 time for calcul the mask position with numpy : 3.1948089599609375e-05 nb_pixel_total : 232 time to create 1 rle with old method : 0.00036025047302246094 length of segment : 15 time for calcul the mask position with numpy : 3.075599670410156e-05 nb_pixel_total : 153 time to create 1 rle with old method : 0.0002231597900390625 length of segment : 15 time for calcul the mask position with numpy : 4.220008850097656e-05 nb_pixel_total : 836 time to create 1 rle with old method : 0.0011000633239746094 length of segment : 44 time for calcul the mask position with numpy : 4.100799560546875e-05 nb_pixel_total : 668 time to create 1 rle with old method : 0.0008919239044189453 length of segment : 25 time for calcul the mask position with numpy : 3.743171691894531e-05 nb_pixel_total : 440 time to create 1 rle with old method : 0.0006322860717773438 length of segment : 18 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 147.19766 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 14 time for calcul the mask position with numpy : 4.9591064453125e-05 nb_pixel_total : 140 time to create 1 rle with old method : 0.0002484321594238281 length of segment : 18 time for calcul the mask position with numpy : 9.298324584960938e-05 nb_pixel_total : 1282 time to create 1 rle with old method : 0.0027954578399658203 length of segment : 59 time for calcul the mask position with numpy : 7.367134094238281e-05 nb_pixel_total : 361 time to create 1 rle with old method : 0.0007991790771484375 length of segment : 25 time for calcul the mask position with numpy : 5.078315734863281e-05 nb_pixel_total : 505 time to create 1 rle with old method : 0.000644683837890625 length of segment : 40 time for calcul the mask position with numpy : 5.817413330078125e-05 nb_pixel_total : 735 time to create 1 rle with old method : 0.0009794235229492188 length of segment : 71 time for calcul the mask position with numpy : 4.363059997558594e-05 nb_pixel_total : 248 time to create 1 rle with old method : 0.00042057037353515625 length of segment : 14 time for calcul the mask position with numpy : 4.267692565917969e-05 nb_pixel_total : 302 time to create 1 rle with old method : 0.00041747093200683594 length of segment : 32 time for calcul the mask position with numpy : 8.678436279296875e-05 nb_pixel_total : 223 time to create 1 rle with old method : 0.00034308433532714844 length of segment : 25 time for calcul the mask position with numpy : 4.410743713378906e-05 nb_pixel_total : 203 time to create 1 rle with old method : 0.0002961158752441406 length of segment : 17 time for calcul the mask position with numpy : 4.172325134277344e-05 nb_pixel_total : 255 time to create 1 rle with old method : 0.0004057884216308594 length of segment : 17 time for calcul the mask position with numpy : 4.100799560546875e-05 nb_pixel_total : 148 time to create 1 rle with old method : 0.00022077560424804688 length of segment : 22 time for calcul the mask position with numpy : 4.0531158447265625e-05 nb_pixel_total : 117 time to create 1 rle with old method : 0.00017786026000976562 length of segment : 25 time for calcul the mask position with numpy : 3.910064697265625e-05 nb_pixel_total : 72 time to create 1 rle with old method : 0.00011873245239257812 length of segment : 17 time for calcul the mask position with numpy : 4.220008850097656e-05 nb_pixel_total : 242 time to create 1 rle with old method : 0.000347137451171875 length of segment : 22 Processing 1 images image shape: (280, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 138.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 2 time for calcul the mask position with numpy : 3.9577484130859375e-05 nb_pixel_total : 20 time to create 1 rle with old method : 0.00011873245239257812 length of segment : 4 time for calcul the mask position with numpy : 5.340576171875e-05 nb_pixel_total : 5 time to create 1 rle with old method : 2.2172927856445312e-05 length of segment : 5 NEW PHOTO pour l'instant on ne peut pas sauvegarder la photo dans les tile Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -115.32891 max: 148.37344 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 7 time for calcul the mask position with numpy : 4.9591064453125e-05 nb_pixel_total : 889 time to create 1 rle with old method : 0.0011973381042480469 length of segment : 36 time for calcul the mask position with numpy : 3.886222839355469e-05 nb_pixel_total : 425 time to create 1 rle with old method : 0.0006084442138671875 length of segment : 22 time for calcul the mask position with numpy : 3.409385681152344e-05 nb_pixel_total : 68 time to create 1 rle with old method : 0.00013566017150878906 length of segment : 22 time for calcul the mask position with numpy : 3.457069396972656e-05 nb_pixel_total : 148 time to create 1 rle with old method : 0.00024271011352539062 length of segment : 36 time for calcul the mask position with numpy : 0.0003399848937988281 nb_pixel_total : 19207 time to create 1 rle with old method : 0.022528886795043945 length of segment : 158 time for calcul the mask position with numpy : 5.412101745605469e-05 nb_pixel_total : 136 time to create 1 rle with old method : 0.0002124309539794922 length of segment : 30 time for calcul the mask position with numpy : 4.5299530029296875e-05 nb_pixel_total : 887 time to create 1 rle with old method : 0.001237630844116211 length of segment : 35 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -119.75469 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 4 time for calcul the mask position with numpy : 4.9114227294921875e-05 nb_pixel_total : 390 time to create 1 rle with old method : 0.0005288124084472656 length of segment : 55 time for calcul the mask position with numpy : 5.555152893066406e-05 nb_pixel_total : 1751 time to create 1 rle with old method : 0.0024874210357666016 length of segment : 33 time for calcul the mask position with numpy : 4.649162292480469e-05 nb_pixel_total : 841 time to create 1 rle with old method : 0.0011086463928222656 length of segment : 39 time for calcul the mask position with numpy : 8.678436279296875e-05 nb_pixel_total : 3311 time to create 1 rle with old method : 0.0044708251953125 length of segment : 66 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -113.43828 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 5 time for calcul the mask position with numpy : 0.00010466575622558594 nb_pixel_total : 6084 time to create 1 rle with old method : 0.007329463958740234 length of segment : 155 time for calcul the mask position with numpy : 5.936622619628906e-05 nb_pixel_total : 790 time to create 1 rle with old method : 0.0011081695556640625 length of segment : 24 time for calcul the mask position with numpy : 0.0001876354217529297 nb_pixel_total : 6237 time to create 1 rle with old method : 0.007735013961791992 length of segment : 98 time for calcul the mask position with numpy : 7.915496826171875e-05 nb_pixel_total : 1651 time to create 1 rle with old method : 0.0022869110107421875 length of segment : 26 time for calcul the mask position with numpy : 6.866455078125e-05 nb_pixel_total : 811 time to create 1 rle with old method : 0.0012199878692626953 length of segment : 22 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -120.72344 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 4 time for calcul the mask position with numpy : 3.8623809814453125e-05 nb_pixel_total : 118 time to create 1 rle with old method : 0.0002167224884033203 length of segment : 14 time for calcul the mask position with numpy : 4.673004150390625e-05 nb_pixel_total : 508 time to create 1 rle with old method : 0.0007951259613037109 length of segment : 26 time for calcul the mask position with numpy : 4.458427429199219e-05 nb_pixel_total : 237 time to create 1 rle with old method : 0.00036787986755371094 length of segment : 22 time for calcul the mask position with numpy : 4.124641418457031e-05 nb_pixel_total : 201 time to create 1 rle with old method : 0.0003104209899902344 length of segment : 24 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 225.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 104.79531 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 2 time for calcul the mask position with numpy : 4.100799560546875e-05 nb_pixel_total : 477 time to create 1 rle with old method : 0.0007128715515136719 length of segment : 18 time for calcul the mask position with numpy : 5.888938903808594e-05 nb_pixel_total : 1067 time to create 1 rle with old method : 0.001432180404663086 length of segment : 34 Processing 1 images image shape: (400, 400, 3) min: 13.00000 max: 205.00000 molded_images shape: (1, 640, 640, 3) min: -87.34063 max: 72.64922 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 0 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -109.29766 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 11 time for calcul the mask position with numpy : 6.413459777832031e-05 nb_pixel_total : 1849 time to create 1 rle with old method : 0.002524852752685547 length of segment : 30 time for calcul the mask position with numpy : 8.845329284667969e-05 nb_pixel_total : 890 time to create 1 rle with old method : 0.0012845993041992188 length of segment : 33 time for calcul the mask position with numpy : 7.772445678710938e-05 nb_pixel_total : 2692 time to create 1 rle with old method : 0.003347158432006836 length of segment : 46 time for calcul the mask position with numpy : 4.696846008300781e-05 nb_pixel_total : 418 time to create 1 rle with old method : 0.0006530284881591797 length of segment : 21 time for calcul the mask position with numpy : 4.315376281738281e-05 nb_pixel_total : 402 time to create 1 rle with old method : 0.00067901611328125 length of segment : 18 time for calcul the mask position with numpy : 3.933906555175781e-05 nb_pixel_total : 381 time to create 1 rle with old method : 0.0005619525909423828 length of segment : 18 time for calcul the mask position with numpy : 4.029273986816406e-05 nb_pixel_total : 383 time to create 1 rle with old method : 0.0005998611450195312 length of segment : 21 time for calcul the mask position with numpy : 5.030632019042969e-05 nb_pixel_total : 680 time to create 1 rle with old method : 0.0009477138519287109 length of segment : 39 time for calcul the mask position with numpy : 5.53131103515625e-05 nb_pixel_total : 778 time to create 1 rle with old method : 0.0010857582092285156 length of segment : 40 time for calcul the mask position with numpy : 0.00013017654418945312 nb_pixel_total : 1333 time to create 1 rle with old method : 0.0019354820251464844 length of segment : 79 time for calcul the mask position with numpy : 4.792213439941406e-05 nb_pixel_total : 449 time to create 1 rle with old method : 0.0007140636444091797 length of segment : 17 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -116.20000 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 8 time for calcul the mask position with numpy : 7.534027099609375e-05 nb_pixel_total : 557 time to create 1 rle with old method : 0.0011603832244873047 length of segment : 33 time for calcul the mask position with numpy : 4.5299530029296875e-05 nb_pixel_total : 990 time to create 1 rle with old method : 0.0013005733489990234 length of segment : 64 time for calcul the mask position with numpy : 4.982948303222656e-05 nb_pixel_total : 857 time to create 1 rle with old method : 0.0014979839324951172 length of segment : 26 time for calcul the mask position with numpy : 3.337860107421875e-05 nb_pixel_total : 161 time to create 1 rle with old method : 0.0002841949462890625 length of segment : 11 time for calcul the mask position with numpy : 4.696846008300781e-05 nb_pixel_total : 754 time to create 1 rle with old method : 0.0011050701141357422 length of segment : 32 time for calcul the mask position with numpy : 4.5299530029296875e-05 nb_pixel_total : 211 time to create 1 rle with old method : 0.0003662109375 length of segment : 17 time for calcul the mask position with numpy : 3.2901763916015625e-05 nb_pixel_total : 212 time to create 1 rle with old method : 0.0003409385681152344 length of segment : 18 time for calcul the mask position with numpy : 6.365776062011719e-05 nb_pixel_total : 556 time to create 1 rle with old method : 0.0009090900421142578 length of segment : 33 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -115.21953 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 8 time for calcul the mask position with numpy : 8.130073547363281e-05 nb_pixel_total : 2057 time to create 1 rle with old method : 0.002846240997314453 length of segment : 123 time for calcul the mask position with numpy : 5.53131103515625e-05 nb_pixel_total : 1272 time to create 1 rle with old method : 0.0017266273498535156 length of segment : 36 time for calcul the mask position with numpy : 3.504753112792969e-05 nb_pixel_total : 95 time to create 1 rle with old method : 0.0001819133758544922 length of segment : 9 time for calcul the mask position with numpy : 5.078315734863281e-05 nb_pixel_total : 1185 time to create 1 rle with old method : 0.0014634132385253906 length of segment : 39 time for calcul the mask position with numpy : 0.00036263465881347656 nb_pixel_total : 17262 time to create 1 rle with old method : 0.020165681838989258 length of segment : 278 time for calcul the mask position with numpy : 4.506111145019531e-05 nb_pixel_total : 827 time to create 1 rle with old method : 0.0010678768157958984 length of segment : 37 time for calcul the mask position with numpy : 3.409385681152344e-05 nb_pixel_total : 324 time to create 1 rle with old method : 0.0005695819854736328 length of segment : 10 time for calcul the mask position with numpy : 4.744529724121094e-05 nb_pixel_total : 1288 time to create 1 rle with old method : 0.0015556812286376953 length of segment : 41 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 246.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 124.92266 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 4 time for calcul the mask position with numpy : 4.982948303222656e-05 nb_pixel_total : 835 time to create 1 rle with old method : 0.0010693073272705078 length of segment : 48 time for calcul the mask position with numpy : 0.00017714500427246094 nb_pixel_total : 4263 time to create 1 rle with old method : 0.0052912235260009766 length of segment : 227 time for calcul the mask position with numpy : 7.605552673339844e-05 nb_pixel_total : 1216 time to create 1 rle with old method : 0.0017008781433105469 length of segment : 114 time for calcul the mask position with numpy : 7.557868957519531e-05 nb_pixel_total : 1701 time to create 1 rle with old method : 0.002222299575805664 length of segment : 167 Processing 1 images image shape: (280, 400, 3) min: 8.00000 max: 204.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 69.78594 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 1 time for calcul the mask position with numpy : 0.0009353160858154297 nb_pixel_total : 106054 time to create 1 rle with old method : 0.12315654754638672 length of segment : 280 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 149.45547 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 16 time for calcul the mask position with numpy : 5.1021575927734375e-05 nb_pixel_total : 267 time to create 1 rle with old method : 0.0003631114959716797 length of segment : 24 time for calcul the mask position with numpy : 3.647804260253906e-05 nb_pixel_total : 454 time to create 1 rle with old method : 0.0007054805755615234 length of segment : 26 time for calcul the mask position with numpy : 7.724761962890625e-05 nb_pixel_total : 2340 time to create 1 rle with old method : 0.0034210681915283203 length of segment : 65 time for calcul the mask position with numpy : 4.553794860839844e-05 nb_pixel_total : 997 time to create 1 rle with old method : 0.0019423961639404297 length of segment : 48 time for calcul the mask position with numpy : 4.076957702636719e-05 nb_pixel_total : 232 time to create 1 rle with old method : 0.0004642009735107422 length of segment : 15 time for calcul the mask position with numpy : 4.649162292480469e-05 nb_pixel_total : 610 time to create 1 rle with old method : 0.0007948875427246094 length of segment : 32 time for calcul the mask position with numpy : 4.458427429199219e-05 nb_pixel_total : 583 time to create 1 rle with old method : 0.0008175373077392578 length of segment : 46 time for calcul the mask position with numpy : 3.9577484130859375e-05 nb_pixel_total : 702 time to create 1 rle with old method : 0.0010216236114501953 length of segment : 33 time for calcul the mask position with numpy : 3.5762786865234375e-05 nb_pixel_total : 409 time to create 1 rle with old method : 0.000537872314453125 length of segment : 25 time for calcul the mask position with numpy : 4.100799560546875e-05 nb_pixel_total : 325 time to create 1 rle with old method : 0.00040793418884277344 length of segment : 34 time for calcul the mask position with numpy : 3.504753112792969e-05 nb_pixel_total : 331 time to create 1 rle with old method : 0.0004684925079345703 length of segment : 26 time for calcul the mask position with numpy : 5.9604644775390625e-05 nb_pixel_total : 1322 time to create 1 rle with old method : 0.0018618106842041016 length of segment : 36 time for calcul the mask position with numpy : 3.600120544433594e-05 nb_pixel_total : 344 time to create 1 rle with old method : 0.0004794597625732422 length of segment : 26 time for calcul the mask position with numpy : 3.719329833984375e-05 nb_pixel_total : 519 time to create 1 rle with old method : 0.0006873607635498047 length of segment : 27 time for calcul the mask position with numpy : 4.4345855712890625e-05 nb_pixel_total : 539 time to create 1 rle with old method : 0.0007240772247314453 length of segment : 27 time for calcul the mask position with numpy : 4.8160552978515625e-05 nb_pixel_total : 1331 time to create 1 rle with old method : 0.001775503158569336 length of segment : 36 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 149.43594 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 12 time for calcul the mask position with numpy : 5.340576171875e-05 nb_pixel_total : 586 time to create 1 rle with old method : 0.0007963180541992188 length of segment : 52 time for calcul the mask position with numpy : 4.4345855712890625e-05 nb_pixel_total : 450 time to create 1 rle with old method : 0.0006432533264160156 length of segment : 19 time for calcul the mask position with numpy : 3.552436828613281e-05 nb_pixel_total : 384 time to create 1 rle with old method : 0.0005388259887695312 length of segment : 23 time for calcul the mask position with numpy : 3.3855438232421875e-05 nb_pixel_total : 258 time to create 1 rle with old method : 0.0004203319549560547 length of segment : 19 time for calcul the mask position with numpy : 4.291534423828125e-05 nb_pixel_total : 839 time to create 1 rle with old method : 0.0010769367218017578 length of segment : 41 time for calcul the mask position with numpy : 4.5299530029296875e-05 nb_pixel_total : 874 time to create 1 rle with old method : 0.0012094974517822266 length of segment : 42 time for calcul the mask position with numpy : 4.315376281738281e-05 nb_pixel_total : 646 time to create 1 rle with old method : 0.0009267330169677734 length of segment : 46 time for calcul the mask position with numpy : 3.266334533691406e-05 nb_pixel_total : 178 time to create 1 rle with old method : 0.0002505779266357422 length of segment : 18 time for calcul the mask position with numpy : 3.5762786865234375e-05 nb_pixel_total : 342 time to create 1 rle with old method : 0.0005137920379638672 length of segment : 19 time for calcul the mask position with numpy : 4.00543212890625e-05 nb_pixel_total : 336 time to create 1 rle with old method : 0.0004601478576660156 length of segment : 39 time for calcul the mask position with numpy : 4.839897155761719e-05 nb_pixel_total : 1198 time to create 1 rle with old method : 0.0015833377838134766 length of segment : 41 time for calcul the mask position with numpy : 4.267692565917969e-05 nb_pixel_total : 649 time to create 1 rle with old method : 0.0008828639984130859 length of segment : 28 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 147.84219 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 16 time for calcul the mask position with numpy : 5.555152893066406e-05 nb_pixel_total : 9 time to create 1 rle with old method : 4.315376281738281e-05 length of segment : 4 time for calcul the mask position with numpy : 3.314018249511719e-05 nb_pixel_total : 234 time to create 1 rle with old method : 0.00036334991455078125 length of segment : 15 time for calcul the mask position with numpy : 3.170967102050781e-05 nb_pixel_total : 201 time to create 1 rle with old method : 0.0002987384796142578 length of segment : 16 time for calcul the mask position with numpy : 3.337860107421875e-05 nb_pixel_total : 275 time to create 1 rle with old method : 0.0003719329833984375 length of segment : 26 time for calcul the mask position with numpy : 3.075599670410156e-05 nb_pixel_total : 131 time to create 1 rle with old method : 0.00019931793212890625 length of segment : 19 time for calcul the mask position with numpy : 3.695487976074219e-05 nb_pixel_total : 437 time to create 1 rle with old method : 0.0005877017974853516 length of segment : 37 time for calcul the mask position with numpy : 4.9591064453125e-05 nb_pixel_total : 1384 time to create 1 rle with old method : 0.001795053482055664 length of segment : 64 time for calcul the mask position with numpy : 3.647804260253906e-05 nb_pixel_total : 354 time to create 1 rle with old method : 0.0005369186401367188 length of segment : 26 time for calcul the mask position with numpy : 4.935264587402344e-05 nb_pixel_total : 1271 time to create 1 rle with old method : 0.0016465187072753906 length of segment : 60 time for calcul the mask position with numpy : 3.7670135498046875e-05 nb_pixel_total : 244 time to create 1 rle with old method : 0.0003902912139892578 length of segment : 15 time for calcul the mask position with numpy : 3.0040740966796875e-05 nb_pixel_total : 64 time to create 1 rle with old method : 0.00010418891906738281 length of segment : 19 time for calcul the mask position with numpy : 4.935264587402344e-05 nb_pixel_total : 1225 time to create 1 rle with old method : 0.001489400863647461 length of segment : 55 time for calcul the mask position with numpy : 3.1948089599609375e-05 nb_pixel_total : 217 time to create 1 rle with old method : 0.00032973289489746094 length of segment : 18 time for calcul the mask position with numpy : 5.745887756347656e-05 nb_pixel_total : 681 time to create 1 rle with old method : 0.0008804798126220703 length of segment : 34 time for calcul the mask position with numpy : 4.649162292480469e-05 nb_pixel_total : 1200 time to create 1 rle with old method : 0.0016925334930419922 length of segment : 24 time for calcul the mask position with numpy : 5.364418029785156e-05 nb_pixel_total : 1522 time to create 1 rle with old method : 0.002000093460083008 length of segment : 24 Processing 1 images image shape: (280, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 138.66250 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 1 time for calcul the mask position with numpy : 4.1484832763671875e-05 nb_pixel_total : 20 time to create 1 rle with old method : 7.033348083496094e-05 length of segment : 4 Detection mask done ! Trying to reset tf kernel 2988050 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 88 tf kernel not reseted sub process len(results) : 204 len(list_Values) 204 None max_time_sub_proc : 3600 parent process len(results) : 0 len(list_Values) 204 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5377 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Catched exception ! Connect or reconnect ! Number saved : None batch 1 Loaded 204 chid ids of type : 4228 Number RLEs to save : 9550 save missing photos in datou_result : time spend for datou_step_exec : 21.863522052764893 time spend to save output : 1.0149321556091309 total time spend for step 1 : 22.878454208374023 step2:brightness Tue Feb 11 15:20:52 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure inside step calcul brightness treat image : temp/1739283629_2987498_1336832842_c54645e049f295a3359e0211d1703e5f.jpg treat image : temp/1739283629_2987498_1336832827_11b1f522276254ad10a2398fc3449040.jpg Inside saveOutput : final : False verbose : 0 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 2 time used for this insertion : 0.011940479278564453 begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 2 time used for this insertion : 0.012022018432617188 save missing photos in datou_result : time spend for datou_step_exec : 0.5232861042022705 time spend to save output : 0.028310060501098633 total time spend for step 2 : 0.5515961647033691 step3:blur_detection Tue Feb 11 15:20:53 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure inside step blur_detection methode: ratio et variance treat image : temp/1739283629_2987498_1336832842_c54645e049f295a3359e0211d1703e5f.jpg resize: (1080, 1920) 1336832842 -7.631204178821959 treat image : temp/1739283629_2987498_1336832827_11b1f522276254ad10a2398fc3449040.jpg resize: (1080, 1920) 1336832827 -7.636932450248696 Inside saveOutput : final : False verbose : 0 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 2 time used for this insertion : 0.011872529983520508 begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 2 time used for this insertion : 0.010712623596191406 save missing photos in datou_result : time spend for datou_step_exec : 1.6104204654693604 time spend to save output : 0.027118921279907227 total time spend for step 3 : 1.6375393867492676 step4:rle_unique_nms_with_priority Tue Feb 11 15:20:55 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms batch 1 Loaded 204 chid ids of type : 4228 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++nb_obj : 105 nb_hashtags : 7 time to prepare the origin masks : 1.0734868049621582 time for calcul the mask position with numpy : 0.022379636764526367 nb_pixel_total : 1769863 time to create 1 rle with new method : 0.04801654815673828 time for calcul the mask position with numpy : 0.00835871696472168 nb_pixel_total : 68420 time to create 1 rle with old method : 0.07820463180541992 time for calcul the mask position with numpy : 0.008507490158081055 nb_pixel_total : 449 time to create 1 rle with old method : 0.0005419254302978516 time for calcul the mask position with numpy : 0.008435964584350586 nb_pixel_total : 14 time to create 1 rle with old method : 3.1948089599609375e-05 time for calcul the mask position with numpy : 0.00837397575378418 nb_pixel_total : 73 time to create 1 rle with old method : 9.298324584960938e-05 time for calcul the mask position with numpy : 0.008418798446655273 nb_pixel_total : 922 time to create 1 rle with old method : 0.0010809898376464844 time for calcul the mask position with numpy : 0.008074522018432617 nb_pixel_total : 145 time to create 1 rle with old method : 0.00016546249389648438 time for calcul the mask position with numpy : 0.008215665817260742 nb_pixel_total : 21 time to create 1 rle with old method : 5.125999450683594e-05 time for calcul the mask position with numpy : 0.008240699768066406 nb_pixel_total : 5384 time to create 1 rle with old method : 0.005997180938720703 time for calcul the mask position with numpy : 0.00939178466796875 nb_pixel_total : 11 time to create 1 rle with old method : 4.124641418457031e-05 time for calcul the mask position with numpy : 0.008954524993896484 nb_pixel_total : 19777 time to create 1 rle with old method : 0.02254796028137207 time for calcul the mask position with numpy : 0.008563518524169922 nb_pixel_total : 318 time to create 1 rle with old method : 0.0004055500030517578 time for calcul the mask position with numpy : 0.008497953414916992 nb_pixel_total : 8 time to create 1 rle with old method : 3.457069396972656e-05 time for calcul the mask position with numpy : 0.008580684661865234 nb_pixel_total : 3 time to create 1 rle with old method : 2.6464462280273438e-05 time for calcul the mask position with numpy : 0.00854945182800293 nb_pixel_total : 211 time to create 1 rle with old method : 0.0003399848937988281 time for calcul the mask position with numpy : 0.008561134338378906 nb_pixel_total : 2317 time to create 1 rle with old method : 0.002815723419189453 time for calcul the mask position with numpy : 0.008492708206176758 nb_pixel_total : 559 time to create 1 rle with old method : 0.0006642341613769531 time for calcul the mask position with numpy : 0.008611917495727539 nb_pixel_total : 784 time to create 1 rle with old method : 0.0009379386901855469 time for calcul the mask position with numpy : 0.008569955825805664 nb_pixel_total : 705 time to create 1 rle with old method : 0.0008635520935058594 time for calcul the mask position with numpy : 0.008454084396362305 nb_pixel_total : 504 time to create 1 rle with old method : 0.0006265640258789062 time for calcul the mask position with numpy : 0.008364439010620117 nb_pixel_total : 1059 time to create 1 rle with old method : 0.0013301372528076172 time for calcul the mask position with numpy : 0.008393287658691406 nb_pixel_total : 2501 time to create 1 rle with old method : 0.002855539321899414 time for calcul the mask position with numpy : 0.008409261703491211 nb_pixel_total : 6260 time to create 1 rle with old method : 0.007349967956542969 time for calcul the mask position with numpy : 0.008380889892578125 nb_pixel_total : 10923 time to create 1 rle with old method : 0.012331724166870117 time for calcul the mask position with numpy : 0.008457660675048828 nb_pixel_total : 94 time to create 1 rle with old method : 0.00020074844360351562 time for calcul the mask position with numpy : 0.008558273315429688 nb_pixel_total : 518 time to create 1 rle with old method : 0.0006303787231445312 time for calcul the mask position with numpy : 0.008571147918701172 nb_pixel_total : 16878 time to create 1 rle with old method : 0.019602060317993164 time for calcul the mask position with numpy : 0.008444786071777344 nb_pixel_total : 852 time to create 1 rle with old method : 0.0009968280792236328 time for calcul the mask position with numpy : 0.00841522216796875 nb_pixel_total : 29 time to create 1 rle with old method : 5.030632019042969e-05 time for calcul the mask position with numpy : 0.008511066436767578 nb_pixel_total : 132 time to create 1 rle with old method : 0.00017261505126953125 time for calcul the mask position with numpy : 0.008417844772338867 nb_pixel_total : 1168 time to create 1 rle with old method : 0.0012907981872558594 time for calcul the mask position with numpy : 0.008427143096923828 nb_pixel_total : 9915 time to create 1 rle with old method : 0.011577844619750977 time for calcul the mask position with numpy : 0.008500337600708008 nb_pixel_total : 903 time to create 1 rle with old method : 0.0010752677917480469 time for calcul the mask position with numpy : 0.008310794830322266 nb_pixel_total : 844 time to create 1 rle with old method : 0.0010027885437011719 time for calcul the mask position with numpy : 0.008232831954956055 nb_pixel_total : 109 time to create 1 rle with old method : 0.0001475811004638672 time for calcul the mask position with numpy : 0.007993936538696289 nb_pixel_total : 2411 time to create 1 rle with old method : 0.002827167510986328 time for calcul the mask position with numpy : 0.00821828842163086 nb_pixel_total : 81 time to create 1 rle with old method : 0.00014495849609375 time for calcul the mask position with numpy : 0.009262323379516602 nb_pixel_total : 33 time to create 1 rle with old method : 8.0108642578125e-05 time for calcul the mask position with numpy : 0.008613109588623047 nb_pixel_total : 406 time to create 1 rle with old method : 0.0004668235778808594 time for calcul the mask position with numpy : 0.008194684982299805 nb_pixel_total : 133 time to create 1 rle with old method : 0.0001728534698486328 time for calcul the mask position with numpy : 0.00807332992553711 nb_pixel_total : 666 time to create 1 rle with old method : 0.0007691383361816406 time for calcul the mask position with numpy : 0.008137226104736328 nb_pixel_total : 3991 time to create 1 rle with old method : 0.004720449447631836 time for calcul the mask position with numpy : 0.008218765258789062 nb_pixel_total : 738 time to create 1 rle with old method : 0.0008981227874755859 time for calcul the mask position with numpy : 0.007965564727783203 nb_pixel_total : 266 time to create 1 rle with old method : 0.00031685829162597656 time for calcul the mask position with numpy : 0.008035898208618164 nb_pixel_total : 2141 time to create 1 rle with old method : 0.0024971961975097656 time for calcul the mask position with numpy : 0.008208990097045898 nb_pixel_total : 697 time to create 1 rle with old method : 0.0007717609405517578 time for calcul the mask position with numpy : 0.008194446563720703 nb_pixel_total : 1051 time to create 1 rle with old method : 0.0011594295501708984 time for calcul the mask position with numpy : 0.008214950561523438 nb_pixel_total : 396 time to create 1 rle with old method : 0.0004892349243164062 time for calcul the mask position with numpy : 0.008300542831420898 nb_pixel_total : 46 time to create 1 rle with old method : 8.177757263183594e-05 time for calcul the mask position with numpy : 0.008965492248535156 nb_pixel_total : 2725 time to create 1 rle with old method : 0.004558563232421875 time for calcul the mask position with numpy : 0.01004934310913086 nb_pixel_total : 1036 time to create 1 rle with old method : 0.0019888877868652344 time for calcul the mask position with numpy : 0.010156631469726562 nb_pixel_total : 918 time to create 1 rle with old method : 0.0015099048614501953 time for calcul the mask position with numpy : 0.010029315948486328 nb_pixel_total : 1339 time to create 1 rle with old method : 0.0026509761810302734 time for calcul the mask position with numpy : 0.012816667556762695 nb_pixel_total : 419 time to create 1 rle with old method : 0.0007164478302001953 time for calcul the mask position with numpy : 0.009736061096191406 nb_pixel_total : 1824 time to create 1 rle with old method : 0.0029871463775634766 time for calcul the mask position with numpy : 0.010507822036743164 nb_pixel_total : 232 time to create 1 rle with old method : 0.0003218650817871094 time for calcul the mask position with numpy : 0.009595632553100586 nb_pixel_total : 105767 time to create 1 rle with old method : 0.11777997016906738 time for calcul the mask position with numpy : 0.008594036102294922 nb_pixel_total : 1172 time to create 1 rle with old method : 0.00139617919921875 time for calcul the mask position with numpy : 0.008530616760253906 nb_pixel_total : 539 time to create 1 rle with old method : 0.0006628036499023438 time for calcul the mask position with numpy : 0.013148069381713867 nb_pixel_total : 3369 time to create 1 rle with old method : 0.004960298538208008 time for calcul the mask position with numpy : 0.008527040481567383 nb_pixel_total : 55 time to create 1 rle with old method : 0.00012063980102539062 time for calcul the mask position with numpy : 0.008319854736328125 nb_pixel_total : 1 time to create 1 rle with old method : 8.58306884765625e-06 time for calcul the mask position with numpy : 0.008370637893676758 nb_pixel_total : 79 time to create 1 rle with old method : 0.00010132789611816406 time for calcul the mask position with numpy : 0.008386611938476562 nb_pixel_total : 5 time to create 1 rle with old method : 1.33514404296875e-05 time for calcul the mask position with numpy : 0.008183479309082031 nb_pixel_total : 140 time to create 1 rle with old method : 0.00017523765563964844 time for calcul the mask position with numpy : 0.008417129516601562 nb_pixel_total : 230 time to create 1 rle with old method : 0.0002970695495605469 time for calcul the mask position with numpy : 0.008370399475097656 nb_pixel_total : 257 time to create 1 rle with old method : 0.00034236907958984375 time for calcul the mask position with numpy : 0.008214950561523438 nb_pixel_total : 14 time to create 1 rle with old method : 4.0531158447265625e-05 time for calcul the mask position with numpy : 0.008117198944091797 nb_pixel_total : 248 time to create 1 rle with old method : 0.00031375885009765625 time for calcul the mask position with numpy : 0.008243560791015625 nb_pixel_total : 389 time to create 1 rle with old method : 0.0004687309265136719 time for calcul the mask position with numpy : 0.008133888244628906 nb_pixel_total : 232 time to create 1 rle with old method : 0.0002932548522949219 time for calcul the mask position with numpy : 0.008131980895996094 nb_pixel_total : 715 time to create 1 rle with old method : 0.0008220672607421875 time for calcul the mask position with numpy : 0.008257389068603516 nb_pixel_total : 35 time to create 1 rle with old method : 5.7220458984375e-05 time for calcul the mask position with numpy : 0.008408784866333008 nb_pixel_total : 417 time to create 1 rle with old method : 0.0005156993865966797 time for calcul the mask position with numpy : 0.008005142211914062 nb_pixel_total : 457 time to create 1 rle with old method : 0.0005414485931396484 time for calcul the mask position with numpy : 0.008129119873046875 nb_pixel_total : 2428 time to create 1 rle with old method : 0.0028073787689208984 time for calcul the mask position with numpy : 0.008143901824951172 nb_pixel_total : 140 time to create 1 rle with old method : 0.00017333030700683594 time for calcul the mask position with numpy : 0.008327007293701172 nb_pixel_total : 302 time to create 1 rle with old method : 0.0003795623779296875 time for calcul the mask position with numpy : 0.008367300033569336 nb_pixel_total : 505 time to create 1 rle with old method : 0.0006134510040283203 time for calcul the mask position with numpy : 0.00838923454284668 nb_pixel_total : 735 time to create 1 rle with old method : 0.0008542537689208984 time for calcul the mask position with numpy : 0.008379459381103516 nb_pixel_total : 1282 time to create 1 rle with old method : 0.0015256404876708984 time for calcul the mask position with numpy : 0.007962703704833984 nb_pixel_total : 241 time to create 1 rle with old method : 0.0003006458282470703 time for calcul the mask position with numpy : 0.008274316787719727 nb_pixel_total : 203 time to create 1 rle with old method : 0.0002465248107910156 time for calcul the mask position with numpy : 0.008150100708007812 nb_pixel_total : 117 time to create 1 rle with old method : 0.00014829635620117188 time for calcul the mask position with numpy : 0.008175849914550781 nb_pixel_total : 425 time to create 1 rle with old method : 0.0005297660827636719 time for calcul the mask position with numpy : 0.008250713348388672 nb_pixel_total : 274 time to create 1 rle with old method : 0.0003299713134765625 time for calcul the mask position with numpy : 0.008038997650146484 nb_pixel_total : 761 time to create 1 rle with old method : 0.0008940696716308594 time for calcul the mask position with numpy : 0.008034944534301758 nb_pixel_total : 148 time to create 1 rle with old method : 0.0001785755157470703 time for calcul the mask position with numpy : 0.008380413055419922 nb_pixel_total : 1143 time to create 1 rle with old method : 0.001329183578491211 time for calcul the mask position with numpy : 0.008156776428222656 nb_pixel_total : 105 time to create 1 rle with old method : 0.0001246929168701172 time for calcul the mask position with numpy : 0.008183479309082031 nb_pixel_total : 365 time to create 1 rle with old method : 0.00041794776916503906 time for calcul the mask position with numpy : 0.008321285247802734 nb_pixel_total : 7 time to create 1 rle with old method : 2.6702880859375e-05 time for calcul the mask position with numpy : 0.00818634033203125 nb_pixel_total : 37 time to create 1 rle with old method : 5.9604644775390625e-05 time for calcul the mask position with numpy : 0.008204936981201172 nb_pixel_total : 302 time to create 1 rle with old method : 0.0003712177276611328 time for calcul the mask position with numpy : 0.008205413818359375 nb_pixel_total : 1382 time to create 1 rle with old method : 0.0016024112701416016 time for calcul the mask position with numpy : 0.008190631866455078 nb_pixel_total : 20 time to create 1 rle with old method : 2.9325485229492188e-05 time for calcul the mask position with numpy : 0.008220195770263672 nb_pixel_total : 371 time to create 1 rle with old method : 0.0004639625549316406 time for calcul the mask position with numpy : 0.008214473724365234 nb_pixel_total : 836 time to create 1 rle with old method : 0.0009670257568359375 time for calcul the mask position with numpy : 0.008313655853271484 nb_pixel_total : 668 time to create 1 rle with old method : 0.0008037090301513672 time for calcul the mask position with numpy : 0.008301973342895508 nb_pixel_total : 153 time to create 1 rle with old method : 0.00018644332885742188 time for calcul the mask position with numpy : 0.008286237716674805 nb_pixel_total : 538 time to create 1 rle with old method : 0.0006270408630371094 time for calcul the mask position with numpy : 0.008383035659790039 nb_pixel_total : 361 time to create 1 rle with old method : 0.0004475116729736328 time for calcul the mask position with numpy : 0.00829315185546875 nb_pixel_total : 664 time to create 1 rle with old method : 0.0007832050323486328 time for calcul the mask position with numpy : 0.008312702178955078 nb_pixel_total : 242 time to create 1 rle with old method : 0.0002903938293457031 time for calcul the mask position with numpy : 0.008126497268676758 nb_pixel_total : 72 time to create 1 rle with old method : 9.083747863769531e-05 time for calcul the mask position with numpy : 0.008050918579101562 nb_pixel_total : 430 time to create 1 rle with old method : 0.0005164146423339844 create new chi : 1.3260633945465088 after preparing all the mask , begin to delete the rle from the crop_hashtag_id => VR 28-11-20 : il faut déplacer cela apres le save_crop_hashtag_ids_obj de la ligne 9514 ! we have 0 chi objets contains the rles time to delete rle : 0.006407260894775391 batch 1 Loaded 106 chid ids of type : 4230 Number RLEs to save : 10188 TO DO : save crop sub photo not yet done ! save time : 0.6208879947662354 nb_obj : 99 nb_hashtags : 6 time to prepare the origin masks : 1.9458630084991455 time for calcul the mask position with numpy : 0.033391714096069336 nb_pixel_total : 1857082 time to create 1 rle with new method : 0.06439352035522461 time for calcul the mask position with numpy : 0.006783246994018555 nb_pixel_total : 6237 time to create 1 rle with old method : 0.007182598114013672 time for calcul the mask position with numpy : 0.006178617477416992 nb_pixel_total : 425 time to create 1 rle with old method : 0.0005373954772949219 time for calcul the mask position with numpy : 0.005972146987915039 nb_pixel_total : 12 time to create 1 rle with old method : 3.7670135498046875e-05 time for calcul the mask position with numpy : 0.005960226058959961 nb_pixel_total : 68 time to create 1 rle with old method : 9.965896606445312e-05 time for calcul the mask position with numpy : 0.005797386169433594 nb_pixel_total : 889 time to create 1 rle with old method : 0.0010042190551757812 time for calcul the mask position with numpy : 0.006051778793334961 nb_pixel_total : 148 time to create 1 rle with old method : 0.00019598007202148438 time for calcul the mask position with numpy : 0.0059773921966552734 nb_pixel_total : 28 time to create 1 rle with old method : 7.939338684082031e-05 time for calcul the mask position with numpy : 0.006010293960571289 nb_pixel_total : 3311 time to create 1 rle with old method : 0.003584623336791992 time for calcul the mask position with numpy : 0.005822896957397461 nb_pixel_total : 201 time to create 1 rle with old method : 0.0002415180206298828 time for calcul the mask position with numpy : 0.006146430969238281 nb_pixel_total : 19207 time to create 1 rle with old method : 0.02068328857421875 time for calcul the mask position with numpy : 0.005995750427246094 nb_pixel_total : 390 time to create 1 rle with old method : 0.0004901885986328125 time for calcul the mask position with numpy : 0.0057528018951416016 nb_pixel_total : 1751 time to create 1 rle with old method : 0.00197601318359375 time for calcul the mask position with numpy : 0.005877256393432617 nb_pixel_total : 237 time to create 1 rle with old method : 0.0002651214599609375 time for calcul the mask position with numpy : 0.006173610687255859 nb_pixel_total : 841 time to create 1 rle with old method : 0.0009539127349853516 time for calcul the mask position with numpy : 0.00577545166015625 nb_pixel_total : 790 time to create 1 rle with old method : 0.0009579658508300781 time for calcul the mask position with numpy : 0.0057468414306640625 nb_pixel_total : 175 time to create 1 rle with old method : 0.0002510547637939453 time for calcul the mask position with numpy : 0.005934953689575195 nb_pixel_total : 508 time to create 1 rle with old method : 0.0006396770477294922 time for calcul the mask position with numpy : 0.005766868591308594 nb_pixel_total : 118 time to create 1 rle with old method : 0.0001666545867919922 time for calcul the mask position with numpy : 0.0058901309967041016 nb_pixel_total : 1651 time to create 1 rle with old method : 0.0019218921661376953 time for calcul the mask position with numpy : 0.006009101867675781 nb_pixel_total : 6084 time to create 1 rle with old method : 0.006905078887939453 time for calcul the mask position with numpy : 0.005978107452392578 nb_pixel_total : 1067 time to create 1 rle with old method : 0.0013222694396972656 time for calcul the mask position with numpy : 0.005930423736572266 nb_pixel_total : 477 time to create 1 rle with old method : 0.0005438327789306641 time for calcul the mask position with numpy : 0.009596824645996094 nb_pixel_total : 324 time to create 1 rle with old method : 0.0004627704620361328 time for calcul the mask position with numpy : 0.0061457157135009766 nb_pixel_total : 161 time to create 1 rle with old method : 0.00022745132446289062 time for calcul the mask position with numpy : 0.006160736083984375 nb_pixel_total : 17262 time to create 1 rle with old method : 0.019789934158325195 time for calcul the mask position with numpy : 0.00634765625 nb_pixel_total : 827 time to create 1 rle with old method : 0.0009646415710449219 time for calcul the mask position with numpy : 0.006026744842529297 nb_pixel_total : 211 time to create 1 rle with old method : 0.0002741813659667969 time for calcul the mask position with numpy : 0.00793766975402832 nb_pixel_total : 19 time to create 1 rle with old method : 8.96453857421875e-05 time for calcul the mask position with numpy : 0.009215354919433594 nb_pixel_total : 857 time to create 1 rle with old method : 0.0016074180603027344 time for calcul the mask position with numpy : 0.00793600082397461 nb_pixel_total : 110 time to create 1 rle with old method : 0.0002167224884033203 time for calcul the mask position with numpy : 0.007917404174804688 nb_pixel_total : 1185 time to create 1 rle with old method : 0.002011537551879883 time for calcul the mask position with numpy : 0.0077838897705078125 nb_pixel_total : 890 time to create 1 rle with old method : 0.0015175342559814453 time for calcul the mask position with numpy : 0.010231733322143555 nb_pixel_total : 671 time to create 1 rle with old method : 0.0013654232025146484 time for calcul the mask position with numpy : 0.01331472396850586 nb_pixel_total : 402 time to create 1 rle with old method : 0.0007457733154296875 time for calcul the mask position with numpy : 0.013448953628540039 nb_pixel_total : 1154 time to create 1 rle with old method : 0.002157449722290039 time for calcul the mask position with numpy : 0.01593947410583496 nb_pixel_total : 1216 time to create 1 rle with old method : 0.0019185543060302734 time for calcul the mask position with numpy : 0.012118101119995117 nb_pixel_total : 4263 time to create 1 rle with old method : 0.004911899566650391 time for calcul the mask position with numpy : 0.010895729064941406 nb_pixel_total : 43 time to create 1 rle with old method : 8.916854858398438e-05 time for calcul the mask position with numpy : 0.010854244232177734 nb_pixel_total : 754 time to create 1 rle with old method : 0.0009632110595703125 time for calcul the mask position with numpy : 0.010618925094604492 nb_pixel_total : 2057 time to create 1 rle with old method : 0.0024671554565429688 time for calcul the mask position with numpy : 0.010568857192993164 nb_pixel_total : 103 time to create 1 rle with old method : 0.0002472400665283203 time for calcul the mask position with numpy : 0.011001110076904297 nb_pixel_total : 680 time to create 1 rle with old method : 0.0008416175842285156 time for calcul the mask position with numpy : 0.010373353958129883 nb_pixel_total : 990 time to create 1 rle with old method : 0.0011949539184570312 time for calcul the mask position with numpy : 0.010571479797363281 nb_pixel_total : 381 time to create 1 rle with old method : 0.0004706382751464844 time for calcul the mask position with numpy : 0.010248184204101562 nb_pixel_total : 557 time to create 1 rle with old method : 0.0007197856903076172 time for calcul the mask position with numpy : 0.01048588752746582 nb_pixel_total : 62 time to create 1 rle with old method : 0.0001361370086669922 time for calcul the mask position with numpy : 0.011576652526855469 nb_pixel_total : 2692 time to create 1 rle with old method : 0.003093242645263672 time for calcul the mask position with numpy : 0.010310649871826172 nb_pixel_total : 835 time to create 1 rle with old method : 0.0010197162628173828 time for calcul the mask position with numpy : 0.006975412368774414 nb_pixel_total : 95 time to create 1 rle with old method : 0.00018906593322753906 time for calcul the mask position with numpy : 0.011734247207641602 nb_pixel_total : 1272 time to create 1 rle with old method : 0.0023012161254882812 time for calcul the mask position with numpy : 0.012752294540405273 nb_pixel_total : 383 time to create 1 rle with old method : 0.0007596015930175781 time for calcul the mask position with numpy : 0.012636661529541016 nb_pixel_total : 1849 time to create 1 rle with old method : 0.0033326148986816406 time for calcul the mask position with numpy : 0.012603282928466797 nb_pixel_total : 418 time to create 1 rle with old method : 0.0007812976837158203 time for calcul the mask position with numpy : 0.012755155563354492 nb_pixel_total : 997 time to create 1 rle with old method : 0.0018169879913330078 time for calcul the mask position with numpy : 0.014388561248779297 nb_pixel_total : 106054 time to create 1 rle with old method : 0.13681983947753906 time for calcul the mask position with numpy : 0.010305166244506836 nb_pixel_total : 131 time to create 1 rle with old method : 0.00017595291137695312 time for calcul the mask position with numpy : 0.01030588150024414 nb_pixel_total : 258 time to create 1 rle with old method : 0.00034165382385253906 time for calcul the mask position with numpy : 0.010376453399658203 nb_pixel_total : 21 time to create 1 rle with old method : 6.771087646484375e-05 time for calcul the mask position with numpy : 0.010219573974609375 nb_pixel_total : 234 time to create 1 rle with old method : 0.00030231475830078125 time for calcul the mask position with numpy : 0.01027989387512207 nb_pixel_total : 384 time to create 1 rle with old method : 0.0004782676696777344 time for calcul the mask position with numpy : 0.010218620300292969 nb_pixel_total : 409 time to create 1 rle with old method : 0.0005438327789306641 time for calcul the mask position with numpy : 0.01012110710144043 nb_pixel_total : 702 time to create 1 rle with old method : 0.0008497238159179688 time for calcul the mask position with numpy : 0.01021718978881836 nb_pixel_total : 450 time to create 1 rle with old method : 0.0005636215209960938 time for calcul the mask position with numpy : 0.006237983703613281 nb_pixel_total : 454 time to create 1 rle with old method : 0.0005447864532470703 time for calcul the mask position with numpy : 0.006264448165893555 nb_pixel_total : 583 time to create 1 rle with old method : 0.0007078647613525391 time for calcul the mask position with numpy : 0.006361246109008789 nb_pixel_total : 437 time to create 1 rle with old method : 0.0005247592926025391 time for calcul the mask position with numpy : 0.006141185760498047 nb_pixel_total : 2340 time to create 1 rle with old method : 0.0028069019317626953 time for calcul the mask position with numpy : 0.006367683410644531 nb_pixel_total : 1384 time to create 1 rle with old method : 0.0016350746154785156 time for calcul the mask position with numpy : 0.006449460983276367 nb_pixel_total : 610 time to create 1 rle with old method : 0.0007371902465820312 time for calcul the mask position with numpy : 0.006315708160400391 nb_pixel_total : 217 time to create 1 rle with old method : 0.0002872943878173828 time for calcul the mask position with numpy : 0.006485462188720703 nb_pixel_total : 1271 time to create 1 rle with old method : 0.0014543533325195312 time for calcul the mask position with numpy : 0.006428718566894531 nb_pixel_total : 325 time to create 1 rle with old method : 0.0004284381866455078 time for calcul the mask position with numpy : 0.00620722770690918 nb_pixel_total : 232 time to create 1 rle with old method : 0.00028061866760253906 time for calcul the mask position with numpy : 0.006259918212890625 nb_pixel_total : 201 time to create 1 rle with old method : 0.0002562999725341797 time for calcul the mask position with numpy : 0.006300926208496094 nb_pixel_total : 27 time to create 1 rle with old method : 5.054473876953125e-05 time for calcul the mask position with numpy : 0.0063321590423583984 nb_pixel_total : 342 time to create 1 rle with old method : 0.00042057037353515625 time for calcul the mask position with numpy : 0.00600433349609375 nb_pixel_total : 267 time to create 1 rle with old method : 0.0003254413604736328 time for calcul the mask position with numpy : 0.006061553955078125 nb_pixel_total : 646 time to create 1 rle with old method : 0.0007336139678955078 time for calcul the mask position with numpy : 0.0059967041015625 nb_pixel_total : 1322 time to create 1 rle with old method : 0.0015234947204589844 time for calcul the mask position with numpy : 0.00617218017578125 nb_pixel_total : 344 time to create 1 rle with old method : 0.00038909912109375 time for calcul the mask position with numpy : 0.005888938903808594 nb_pixel_total : 275 time to create 1 rle with old method : 0.0003437995910644531 time for calcul the mask position with numpy : 0.00579524040222168 nb_pixel_total : 66 time to create 1 rle with old method : 0.00012254714965820312 time for calcul the mask position with numpy : 0.005692958831787109 nb_pixel_total : 1200 time to create 1 rle with old method : 0.0014340877532958984 time for calcul the mask position with numpy : 0.005896329879760742 nb_pixel_total : 331 time to create 1 rle with old method : 0.0004191398620605469 time for calcul the mask position with numpy : 0.005951404571533203 nb_pixel_total : 1198 time to create 1 rle with old method : 0.0015499591827392578 time for calcul the mask position with numpy : 0.005959987640380859 nb_pixel_total : 9 time to create 1 rle with old method : 3.3855438232421875e-05 time for calcul the mask position with numpy : 0.005913734436035156 nb_pixel_total : 20 time to create 1 rle with old method : 4.1961669921875e-05 time for calcul the mask position with numpy : 0.00581669807434082 nb_pixel_total : 839 time to create 1 rle with old method : 0.0009565353393554688 time for calcul the mask position with numpy : 0.005793094635009766 nb_pixel_total : 649 time to create 1 rle with old method : 0.0007679462432861328 time for calcul the mask position with numpy : 0.005725860595703125 nb_pixel_total : 178 time to create 1 rle with old method : 0.00020766258239746094 time for calcul the mask position with numpy : 0.005820512771606445 nb_pixel_total : 681 time to create 1 rle with old method : 0.0008175373077392578 time for calcul the mask position with numpy : 0.005877256393432617 nb_pixel_total : 519 time to create 1 rle with old method : 0.0005791187286376953 time for calcul the mask position with numpy : 0.005867719650268555 nb_pixel_total : 28 time to create 1 rle with old method : 6.508827209472656e-05 time for calcul the mask position with numpy : 0.005732059478759766 nb_pixel_total : 354 time to create 1 rle with old method : 0.0003981590270996094 time for calcul the mask position with numpy : 0.006319284439086914 nb_pixel_total : 586 time to create 1 rle with old method : 0.00061798095703125 time for calcul the mask position with numpy : 0.005860090255737305 nb_pixel_total : 874 time to create 1 rle with old method : 0.0010685920715332031 time for calcul the mask position with numpy : 0.005799293518066406 nb_pixel_total : 336 time to create 1 rle with old method : 0.00038886070251464844 time for calcul the mask position with numpy : 0.005912065505981445 nb_pixel_total : 64 time to create 1 rle with old method : 0.00014257431030273438 time for calcul the mask position with numpy : 0.00591731071472168 nb_pixel_total : 331 time to create 1 rle with old method : 0.0003943443298339844 create new chi : 1.1463043689727783 after preparing all the mask , begin to delete the rle from the crop_hashtag_id => VR 28-11-20 : il faut déplacer cela apres le save_crop_hashtag_ids_obj de la ligne 9514 ! we have 0 chi objets contains the rles time to delete rle : 0.0011775493621826172 batch 1 Loaded 100 chid ids of type : 4230 Number RLEs to save : 9217 TO DO : save crop sub photo not yet done ! save time : 0.5916166305541992 map_output_result : {1336832842: (0.0, 'Should be the crop_list due to order', 0.0), 1336832827: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1336832842, 1336832827] Looping around the photos to save general results len do output : 2 /1336832842.Didn't retrieve data . /1336832827.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2576510') ('4234', '20447187', '1336832842', None, None, None, None, None, '2576510') ('4234', None, None, None, None, None, None, None, '2576510') ('4234', '20447187', '1336832827', None, None, None, None, None, '2576510') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 6 time used for this insertion : 0.01142263412475586 save_final save missing photos in datou_result : time spend for datou_step_exec : 7.099421501159668 time spend to save output : 0.01167917251586914 total time spend for step 4 : 7.111100673675537 step5:crop_condition Tue Feb 11 15:21:02 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 4230 Loading chi in step crop for list_pids : 2 ! batch 1 Loaded 206 chid ids of type : 4230 begin to crop the class : papier param for this class : {'min_score': 0.6} filtre for class : papier hashtag_id of this class : 492668766 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 71 About to insert : list_path_to_insert length 71 new photo from crops ! About to upload 71 photos upload in portfolio : 4869462 init cache_photo without model_param we have 71 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739283664_2987498 we have uploaded 71 photos in the portfolio 4869462 time of upload the photos Elapsed time : 29.933412790298462 we have finished the crop for the class : papier begin to crop the class : carton param for this class : {'min_score': 0.6} filtre for class : carton hashtag_id of this class : 492774966 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 29 About to insert : list_path_to_insert length 29 new photo from crops ! About to upload 29 photos upload in portfolio : 4869462 init cache_photo without model_param we have 29 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739283694_2987498 we have uploaded 29 photos in the portfolio 4869462 time of upload the photos Elapsed time : 7.361148118972778 we have finished the crop for the class : carton begin to crop the class : metal param for this class : {'min_score': 0.6} filtre for class : metal hashtag_id of this class : 492628673 begin to crop the class : pet_clair param for this class : {'min_score': 0.6} filtre for class : pet_clair hashtag_id of this class : 2107755846 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! About to upload 9 photos upload in portfolio : 4869462 init cache_photo without model_param we have 9 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739283702_2987498 we have uploaded 9 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.0703232288360596 we have finished the crop for the class : pet_clair begin to crop the class : autre param for this class : {'min_score': 0.6} filtre for class : autre hashtag_id of this class : 494826614 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 20 About to insert : list_path_to_insert length 20 new photo from crops ! About to upload 20 photos upload in portfolio : 4869462 init cache_photo without model_param we have 20 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739283704_2987498 we have uploaded 20 photos in the portfolio 4869462 time of upload the photos Elapsed time : 4.803578853607178 we have finished the crop for the class : autre begin to crop the class : pehd param for this class : {'min_score': 0.6} filtre for class : pehd hashtag_id of this class : 628944319 begin to crop the class : pet_fonce param for this class : {'min_score': 0.6} filtre for class : pet_fonce hashtag_id of this class : 2107755900 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1336832842, 1336832827] Looping around the photos to save general results len do output : 129 /1336845363Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845364Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845365Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845366Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845367Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845368Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845369Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845370Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845371Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845372Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845374Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845375Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845376Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845377Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845378Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845379Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845380Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845381Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845382Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845383Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845384Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845385Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845386Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845387Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845388Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845389Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845390Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845391Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845392Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845393Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845394Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845395Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845396Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845397Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845398Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845399Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845400Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845401Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845402Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845403Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845404Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845405Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845406Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845407Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845408Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845409Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845410Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845411Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845412Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845413Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845414Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845415Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845416Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845417Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845418Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845419Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845420Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845421Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845423Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845424Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845425Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845426Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845427Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845428Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845429Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845430Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845431Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845432Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845433Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845434Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845435Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845487Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845488Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845489Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845492Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845493Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845494Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845495Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845496Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845497Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845498Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845499Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845500Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845502Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845503Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845504Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845505Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845506Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845507Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845508Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845509Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845510Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845511Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845512Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845513Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845514Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845515Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845516Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845517Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845519Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845522Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845523Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845524Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845525Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845526Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845527Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845528Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845529Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845530Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845535Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845536Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845537Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845538Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845539Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845540Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845541Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845542Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845543Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845544Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845545Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845546Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845547Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845548Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845549Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845551Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845553Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845554Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845555Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336845556Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2576510') ('4234', '20447187', '1336832842', None, None, None, None, None, '2576510') ('4234', None, None, None, None, None, None, None, '2576510') ('4234', '20447187', '1336832827', None, None, None, None, None, '2576510') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 389 time used for this insertion : 0.03242778778076172 save_final save missing photos in datou_result : time spend for datou_step_exec : 47.152382373809814 time spend to save output : 0.03740048408508301 total time spend for step 5 : 47.1897828578949 step6:thcl Tue Feb 11 15:21:49 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! we are using the classfication for only one thcl 3237 time to import caffe and check if the image exist : 0.008346796035766602 time to convert the images to numpy array : 0.03519892692565918 time to import caffe and check if the image exist : 0.004729270935058594 time to convert the images to numpy array : 0.03846859931945801 time to import caffe and check if the image exist : 0.007226467132568359 time to convert the images to numpy array : 0.04183363914489746 time to import caffe and check if the image exist : 0.009751081466674805 time to convert the images to numpy array : 0.04191994667053223 time to import caffe and check if the image exist : 0.008231639862060547 time to convert the images to numpy array : 0.0440213680267334 time to import caffe and check if the image exist : 0.01521754264831543 time to convert the images to numpy array : 0.03867936134338379 time to import caffe and check if the image exist : 0.009046554565429688 time to convert the images to numpy array : 0.045432329177856445 time to import caffe and check if the image exist : 0.01405191421508789 time to convert the images to numpy array : 0.041577816009521484 time to import caffe and check if the image exist : 0.02359747886657715 time to convert the images to numpy array : 0.03458833694458008 time to import caffe and check if the image exist : 0.007284402847290039 time to convert the images to numpy array : 0.050534963607788086 total time to convert the images to numpy array : 0.2703824043273926 list photo_ids error: [] list photo_ids correct : [1336845430, 1336845431, 1336845432, 1336845433, 1336845434, 1336845435, 1336845487, 1336845488, 1336845489, 1336845492, 1336845493, 1336845494, 1336845495, 1336845403, 1336845404, 1336845405, 1336845406, 1336845407, 1336845408, 1336845409, 1336845410, 1336845411, 1336845412, 1336845413, 1336845414, 1336845415, 1336845363, 1336845364, 1336845365, 1336845366, 1336845367, 1336845368, 1336845369, 1336845370, 1336845371, 1336845372, 1336845374, 1336845375, 1336845376, 1336845510, 1336845511, 1336845512, 1336845513, 1336845514, 1336845515, 1336845516, 1336845517, 1336845519, 1336845522, 1336845523, 1336845524, 1336845525, 1336845377, 1336845378, 1336845379, 1336845380, 1336845381, 1336845382, 1336845383, 1336845384, 1336845385, 1336845386, 1336845387, 1336845388, 1336845389, 1336845416, 1336845417, 1336845418, 1336845419, 1336845420, 1336845421, 1336845423, 1336845424, 1336845425, 1336845426, 1336845427, 1336845428, 1336845429, 1336845543, 1336845544, 1336845545, 1336845546, 1336845547, 1336845548, 1336845549, 1336845551, 1336845553, 1336845554, 1336845555, 1336845556, 1336845526, 1336845527, 1336845528, 1336845529, 1336845530, 1336845535, 1336845536, 1336845537, 1336845538, 1336845539, 1336845540, 1336845541, 1336845542, 1336845496, 1336845497, 1336845498, 1336845499, 1336845500, 1336845502, 1336845503, 1336845504, 1336845505, 1336845506, 1336845507, 1336845508, 1336845509, 1336845390, 1336845391, 1336845392, 1336845393, 1336845394, 1336845395, 1336845396, 1336845397, 1336845398, 1336845399, 1336845400, 1336845401, 1336845402] number of photos to traite : 129 try to delete the photos incorrect in DB tagging for thcl : 3237 To do loadFromThcl(), then load ParamDescType : thcl3237 thcls : [{'id': 3237, 'mtr_user_id': 31, 'name': 'learn_rubbia_refus_2500', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton,Film_plastique,PEHD,PET_clair,PET_fonce,Papier,Tetrapak,flou,mal_croppe,metal,refus', 'svm_portfolios_learning': '4865689,4865690,4865686,4865684,4865685,4865688,4865691,4865693,4865692,4865687,4865683', 'photo_hashtag_type': 4158, 'photo_desc_type': 5561, 'type_classification': 'caffe', 'hashtag_id_list': '492774966,2107756122,628944319,2107755846,2107755900,492668766,609991870,492777938,2107755527,492628673,538914404'}] thcl {'id': 3237, 'mtr_user_id': 31, 'name': 'learn_rubbia_refus_2500', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton,Film_plastique,PEHD,PET_clair,PET_fonce,Papier,Tetrapak,flou,mal_croppe,metal,refus', 'svm_portfolios_learning': '4865689,4865690,4865686,4865684,4865685,4865688,4865691,4865693,4865692,4865687,4865683', 'photo_hashtag_type': 4158, 'photo_desc_type': 5561, 'type_classification': 'caffe', 'hashtag_id_list': '492774966,2107756122,628944319,2107755846,2107755900,492668766,609991870,492777938,2107755527,492628673,538914404'} Update svm_hashtag_type_desc : 5561 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5561, 'learn_rubbia_refus_2500', 2048, 2048, 'learn_rubbia_refus_2500', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2021, 12, 2, 19, 10, 8), datetime.datetime(2021, 12, 2, 19, 10, 8)) To loadFromThcl() : net_5561 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10553 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5561, 'learn_rubbia_refus_2500', 2048, 2048, 'learn_rubbia_refus_2500', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2021, 12, 2, 19, 10, 8), datetime.datetime(2021, 12, 2, 19, 10, 8)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_rubbia_refus_2500 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_rubbia_refus_2500 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_rubbia_refus_2500 /data/models_weight/learn_rubbia_refus_2500/caffemodel size_local : 94358479 size in s3 : 94358479 create time local : 2021-12-03 18:29:39 create time in s3 : 2021-12-02 17:49:16 caffemodel already exist and didn't need to update /data/models_weight/learn_rubbia_refus_2500/deploy.prototxt size_local : 32544 size in s3 : 32544 create time local : 2021-12-03 18:29:39 create time in s3 : 2021-12-02 17:49:15 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_rubbia_refus_2500/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-12-03 18:29:40 create time in s3 : 2021-12-02 18:09:52 mean.npy already exist and didn't need to update /data/models_weight/learn_rubbia_refus_2500/synset_words.txt size_local : 334 size in s3 : 334 create time local : 2021-12-03 18:29:40 create time in s3 : 2021-12-02 18:10:06 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_rubbia_refus_2500/deploy.prototxt caffemodel_filename : /data/models_weight/learn_rubbia_refus_2500/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10334 max_wait_temp : 1 max_wait : 0 dict_keys(['pool5', 'prob']) time used to do the prepocess of the images : 0.6121659278869629 time used to do the prediction : 0.5364336967468262 save descriptor for thcl : 3237 time to traite the descriptors : 0.789168119430542 Catched exception ! Connect or reconnect ! storage_type for insertDescriptorsMulti : 3 To insert : 1336845430 To insert : 1336845431 To insert : 1336845432 To insert : 1336845433 To insert : 1336845434 To insert : 1336845435 To insert : 1336845487 To insert : 1336845488 To insert : 1336845489 To insert : 1336845492 To insert : 1336845493 To insert : 1336845494 To insert : 1336845495 To insert : 1336845403 To insert : 1336845404 To insert : 1336845405 To insert : 1336845406 To insert : 1336845407 To insert : 1336845408 To insert : 1336845409 To insert : 1336845410 To insert : 1336845411 To insert : 1336845412 To insert : 1336845413 To insert : 1336845414 To insert : 1336845415 To insert : 1336845363 To insert : 1336845364 To insert : 1336845365 To insert : 1336845366 To insert : 1336845367 To insert : 1336845368 To insert : 1336845369 To insert : 1336845370 To insert : 1336845371 To insert : 1336845372 To insert : 1336845374 To insert : 1336845375 To insert : 1336845376 To insert : 1336845510 To insert : 1336845511 To insert : 1336845512 To insert : 1336845513 To insert : 1336845514 To insert : 1336845515 To insert : 1336845516 To insert : 1336845517 To insert : 1336845519 To insert : 1336845522 To insert : 1336845523 To insert : 1336845524 To insert : 1336845525 To insert : 1336845377 To insert : 1336845378 To insert : 1336845379 To insert : 1336845380 To insert : 1336845381 To insert : 1336845382 To insert : 1336845383 To insert : 1336845384 To insert : 1336845385 To insert : 1336845386 To insert : 1336845387 To insert : 1336845388 To insert : 1336845389 To insert : 1336845416 To insert : 1336845417 To insert : 1336845418 To insert : 1336845419 To insert : 1336845420 To insert : 1336845421 To insert : 1336845423 To insert : 1336845424 To insert : 1336845425 To insert : 1336845426 To insert : 1336845427 To insert : 1336845428 To insert : 1336845429 To insert : 1336845543 To insert : 1336845544 To insert : 1336845545 To insert : 1336845546 To insert : 1336845547 To insert : 1336845548 To insert : 1336845549 To insert : 1336845551 To insert : 1336845553 To insert : 1336845554 To insert : 1336845555 To insert : 1336845556 To insert : 1336845526 To insert : 1336845527 To insert : 1336845528 To insert : 1336845529 To insert : 1336845530 To insert : 1336845535 To insert : 1336845536 To insert : 1336845537 To insert : 1336845538 To insert : 1336845539 To insert : 1336845540 To insert : 1336845541 To insert : 1336845542 To insert : 1336845496 To insert : 1336845497 To insert : 1336845498 To insert : 1336845499 To insert : 1336845500 To insert : 1336845502 To insert : 1336845503 To insert : 1336845504 To insert : 1336845505 To insert : 1336845506 To insert : 1336845507 To insert : 1336845508 To insert : 1336845509 To insert : 1336845390 To insert : 1336845391 To insert : 1336845392 To insert : 1336845393 To insert : 1336845394 To insert : 1336845395 To insert : 1336845396 To insert : 1336845397 To insert : 1336845398 To insert : 1336845399 To insert : 1336845400 To insert : 1336845401 To insert : 1336845402 time to insert the descriptors : 23.109177350997925 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos SAVE THCL : begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 192 time used for this insertion : 0.018542051315307617 save missing photos in datou_result : time spend for datou_step_exec : 28.994476556777954 time spend to save output : 0.06443524360656738 total time spend for step 6 : 29.05891180038452 step7:ventilate_hashtags_in_portfolio Tue Feb 11 15:22:18 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! Iterating over portfolio : 20447187 get user id for portfolio 20447187 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20447187 AND mptpi.`type`=4230 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('flou','pet_fonce','Papier','Tetrapak','papier','Film_plastique','Carton','refus','environnement','carton','PET_clair','pet_clair','metal','PEHD','pehd','mal_croppe','PET_fonce','autre')) AND mptpi.`min_score`=0.6 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20447187 AND mptpi.`type`=4230 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('flou','pet_fonce','Papier','Tetrapak','papier','Film_plastique','Carton','refus','environnement','carton','PET_clair','pet_clair','metal','PEHD','pehd','mal_croppe','PET_fonce','autre')) AND mptpi.`min_score`=0.6 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20447187 AND mptpi.`type`=4231 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('flou','pet_fonce','Papier','Tetrapak','papier','Film_plastique','Carton','refus','environnement','carton','PET_clair','pet_clair','metal','PEHD','pehd','mal_croppe','PET_fonce','autre')) AND mptpi.`min_score`=0.6 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20447187 AND mptpi.`type`=4231 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('flou','pet_fonce','Papier','Tetrapak','papier','Film_plastique','Carton','refus','environnement','carton','PET_clair','pet_clair','metal','PEHD','pehd','mal_croppe','PET_fonce','autre')) AND mptpi.`min_score`=0.6 To do lien utilise dans velours : https://www.fotonower.com/velours/20447818,20447834,20447822,20447821,20447823,20447827,20447825,20447826,20447829,20447830,20447832,20447833,20447835?tags=flou,pet_fonce,papier,tetrapak,film_plastique,carton,refus,environnement,pet_clair,metal,pehd,mal_croppe,autre&datou_id_consolidate=4235&port_consolidate=20447187 Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1336832842, 1336832827] Looping around the photos to save general results len do output : 1 /20447187. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2576510') ('4234', '20447187', '1336832842', None, None, None, None, None, '2576510') ('4234', None, None, None, None, None, None, None, '2576510') ('4234', '20447187', '1336832827', None, None, None, None, None, '2576510') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 time used for this insertion : 0.013257503509521484 save_final save missing photos in datou_result : time spend for datou_step_exec : 2.282416820526123 time spend to save output : 0.01348257064819336 total time spend for step 7 : 2.2958993911743164 step8:final Tue Feb 11 15:22:20 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1336832842: ('0.19305841727143705',), 1336832827: ('0.19305841727143705',)} new output for save of step final : {1336832842: ('0.19305841727143705',), 1336832827: ('0.19305841727143705',)} [1336832842, 1336832827] Looping around the photos to save general results len do output : 2 /1336832842.Didn't retrieve data . /1336832827.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2576510') ('4234', '20447187', '1336832842', None, None, None, None, None, '2576510') ('4234', None, None, None, None, None, None, None, '2576510') ('4234', '20447187', '1336832827', None, None, None, None, None, '2576510') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 6 time used for this insertion : 0.012624740600585938 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.11304140090942383 time spend to save output : 0.012810707092285156 total time spend for step 8 : 0.12585210800170898 step9:velours_tree Tue Feb 11 15:22:20 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure list_portfolios : 20447818,20447834,20447822,20447821,20447823,20447827,20447825,20447826,20447829,20447830,20447832,20447833,20447835 photo desc type : 5561 - Retrieving photos to tag... query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20447818 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20447834 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20447822 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20447821 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20447823 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20447827 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20447825 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20447826 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20447829 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20447830 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20447832 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20447833 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20447835 ORDER BY ph.size desc - Loading descriptors... Size : 2048 len(descriptors) : 129 Compute structured hierarchical clustering... ward : AgglomerativeClustering(n_clusters=6) ward.labels_ : [0 1 3 3 1 4 4 1 0 3 1 1 0 3 0 3 3 3 0 3 3 1 5 0 3 2 3 1 2 1 2 1 1 0 3 1 0 1 1 3 1 4 1 1 3 3 3 0 3 3 3 3 4 3 3 3 5 0 2 1 2 3 1 2 1 1 1 1 0 0 3 3 3 0 0 4 1 0 3 1 4 0 3 3 1 0 3 0 3 3 0 0 4 1 0 3 2 0 0 0 0 1 3 1 0 1 3 1 4 1 1 1 1 1 1 1 3 1 1 1 1 1 1 1 1 1 1 0 3] Elapsed time: 0.01767563819885254 graph_id used : 77914 - Beta version, working pretty good on 11-5-16 ! https://www.fotonower.com/velours/20447818,20447834,20447822,20447821,20447823,20447827,20447825,20447826,20447829,20447830,20447832,20447833,20447835?tags=flou,pet_fonce,papier,tetrapak,film_plastique,carton,refus,environnement,pet_clair,metal,pehd,mal_croppe,autre&datou_id_consolidate=4235&port_consolidate=20447187&tree_id=77914 Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : velours_tree we use saveGeneral [1336832842, 1336832827] Looping around the photos to save general results len do output : 1 /20447187Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2576510') ('4234', '20447187', '1336832842', None, None, None, None, None, '2576510') ('4234', None, None, None, None, None, None, None, '2576510') ('4234', '20447187', '1336832827', None, None, None, None, None, '2576510') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 time used for this insertion : 0.012258768081665039 save_final save missing photos in datou_result : time spend for datou_step_exec : 8.443250179290771 time spend to save output : 0.012538433074951172 total time spend for step 9 : 8.455788612365723 step10:send_mail_cod Tue Feb 11 15:22:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Qualipapia_P20447187_11-02-2025_15_22_29.pdf 20447800 imagette204478001739283749 20447816 imagette204478161739283749 20447804 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette204478041739283749 20447803 imagette204478031739283750 20447805 imagette204478051739283750 20447809 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette204478091739283750 20447807 imagette204478071739283751 20447811 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette204478111739283751 20447812 imagette204478121739283752 20447814 imagette204478141739283752 20447815 imagette204478151739283752 20447817 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette204478171739283752 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20447187 and hashtag_type = 4230 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20447818,20447834,20447822,20447821,20447823,20447827,20447825,20447826,20447829,20447830,20447832,20447833,20447835?tags=flou,pet_fonce,papier,tetrapak,film_plastique,carton,refus,environnement,pet_clair,metal,pehd,mal_croppe,autre&datou_id_consolidate=4235&port_consolidate=20447187&tree_id=77914 your option no_mail is active, we will not send the real mail to your client args[1336832842] : ((1336832842, -7.631204178821959, 492609224), (1336832842, -0.46219101123071094, 496442774), '0.19305841727143705') apple ((1336832842, -7.631204178821959, 492609224), (1336832842, -0.46219101123071094, 496442774), '0.19305841727143705') We are sending mail with results at cod@fotonower.com args[1336832827] : ((1336832827, -7.636932450248696, 492609224), (1336832827, -0.45310740003230404, 496442774), '0.19305841727143705') apple ((1336832827, -7.636932450248696, 492609224), (1336832827, -0.45310740003230404, 496442774), '0.19305841727143705') We are sending mail with results at cod@fotonower.com refus_total : 0.19305841727143705 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20447187 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20447187_11-02-2025_15_22_29.pdf results_Qualipapia_P20447187_11-02-2025_15_22_29.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20447187_11-02-2025_15_22_29.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('4234','20447187','results_Qualipapia_P20447187_11-02-2025_15_22_29.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20447187_11-02-2025_15_22_29.pdf','pdf','','0.14','0.19305841727143705') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1336832842, 1336832827] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2576510') ('4234', '20447187', '1336832842', None, None, None, None, None, '2576510') ('4234', None, None, None, None, None, None, None, '2576510') ('4234', '20447187', '1336832827', None, None, None, None, None, '2576510') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 2 time used for this insertion : 0.013111591339111328 save_final save missing photos in datou_result : time spend for datou_step_exec : 5.224094867706299 time spend to save output : 0.013297796249389648 total time spend for step 10 : 5.2373926639556885 step11:split_time_score Tue Feb 11 15:22:34 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 3442, 'mtr_user_id': 31, 'name': 'classifieur_2camions_valcor_021122_v1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'deux_camions,camion_droite,camion_gauche,pas_de_camion', 'svm_portfolios_learning': '7659379,7659034,7657685,7657114', 'photo_hashtag_type': 4458, 'photo_desc_type': 5723, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107760533,2107760534,2107760535,2107760536'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('12', 2),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 11022025 20447187 Nombre de photos uploadées : 2 / 23040 (0%) 11022025 20447187 Nombre de photos taguées (types de déchets): 2 / 2 (100%) 11022025 20447187 Nombre de photos taguées (volume) : 0 / 2 (0%) elapsed_time : load_data_split_time_score 2.86102294921875e-06 elapsed_time : order_list_meta_photo_and_scores 7.62939453125e-06 ?? elapsed_time : fill_and_build_computed_from_old_data 0.00021529197692871094 elapsed_time : insert_dashboard_record_day_entry 0.03864574432373047 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.19655717368005024 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20440527_11-02-2025_11_19_12.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20440527 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20440527 AND mptpi.`type`=4230 To do Qualite : 0.2739191087147952 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20442838_11-02-2025_12_18_03.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20442838 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20442838 AND mptpi.`type`=4230 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20444319 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20444320 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20444321 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20444322 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20444323 order by id desc limit 1 Qualite : 0.18499769895532558 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20444324_11-02-2025_13_48_56.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20444324 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20444324 AND mptpi.`type`=4230 To do Qualite : 0.20952740655524157 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20444325_11-02-2025_13_49_43.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20444325 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20444325 AND mptpi.`type`=4230 To do Qualite : 0.20295017305943217 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20445087_11-02-2025_14_23_58.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20445087 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20445087 AND mptpi.`type`=4230 To do Qualite : 0.21338363403320454 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20445088_11-02-2025_14_18_55.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20445088 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20445088 AND mptpi.`type`=4230 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20445877 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20445879 order by id desc limit 1 Qualite : 0.14219345588734875 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20445881_11-02-2025_14_47_30.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20445881 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20445881 AND mptpi.`type`=4230 To do Qualite : 0.24626948979078805 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20447185_11-02-2025_15_18_56.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20447185 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20447185 AND mptpi.`type`=4230 To do Qualite : 0.19305841727143705 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20447187_11-02-2025_15_22_29.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20447187 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20447187 AND mptpi.`type`=4230 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'11022025': {'nb_upload': 2, 'nb_taggue_class': 2, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1336832842, 1336832827] Looping around the photos to save general results len do output : 1 /20447187Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2576510') ('4234', '20447187', '1336832842', None, None, None, None, None, '2576510') ('4234', None, None, None, None, None, None, None, '2576510') ('4234', '20447187', '1336832827', None, None, None, None, None, '2576510') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 time used for this insertion : 0.02907109260559082 save_final save missing photos in datou_result : time spend for datou_step_exec : 49.725276708602905 time spend to save output : 0.029267072677612305 total time spend for step 11 : 49.75454378128052 caffe_path_current : About to save ! 2 After save, about to update current ! ret : 1 len(input) + len(total_photo_id_missing) : 2 set_done_treatment 31.66user 20.15system 2:59.20elapsed 28%CPU (0avgtext+0avgdata 2463064maxresident)k 94648inputs+10776outputs (49major+2034887minor)pagefaults 0swaps