python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 4234 -l 5 ' -s datou_current_4234 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3748744 load datou : 4234 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5275, 'learn_RUBBIA_REFUS_AMIENS_23', 16384, 25088, 'learn_RUBBIA_REFUS_AMIENS_23', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 23, 14, 19, 39), datetime.datetime(2021, 4, 23, 14, 19, 39)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5561, 'learn_rubbia_refus_2500', 2048, 2048, 'learn_rubbia_refus_2500', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2021, 12, 2, 19, 10, 8), datetime.datetime(2021, 12, 2, 19, 10, 8)) load thcls load THCL from format json or kwargs add thcl : 2847 in CacheModelConfig load THCL from format json or kwargs add thcl : 3237 in CacheModelConfig load pdts add pdt : 5275 in CacheModelConfig add pdt : 5561 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 list_input_json: [] Current got : datou_id : 4234, datou_cur_ids : ['2564355'] with mtr_portfolio_ids : ['20334353'] and first list_photo_ids : [] new path : /proc/3748744/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, brightness, blur_detection, rle_unique_nms_with_priority, crop_condition, thcl, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , BFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 2 ; length of list_pids : 2 ; length of list_args : 2 time to download the photos : 0.27546238899230957 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Fri Feb 7 13:20:28 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10774 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-07 13:20:31.566661: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-07 13:20:31.599133: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-07 13:20:31.601158: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fba94000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-07 13:20:31.601189: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-07 13:20:31.606371: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-07 13:20:31.863518: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x30ce1720 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-07 13:20:31.863598: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-07 13:20:31.865007: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-07 13:20:31.865872: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-07 13:20:31.870526: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-07 13:20:31.884914: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-07 13:20:31.886285: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-07 13:20:31.917983: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-07 13:20:31.923128: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-07 13:20:31.970674: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-07 13:20:31.972539: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-07 13:20:31.972634: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-07 13:20:31.973583: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-07 13:20:31.973603: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-07 13:20:31.973614: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-07 13:20:31.975313: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9985 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-07 13:20:32.303914: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-07 13:20:32.304007: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-07 13:20:32.304024: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-07 13:20:32.304040: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-07 13:20:32.304054: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-07 13:20:32.304068: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-07 13:20:32.304083: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-07 13:20:32.304098: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-07 13:20:32.305416: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-07 13:20:32.306628: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-07 13:20:32.306657: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-07 13:20:32.306672: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-07 13:20:32.306686: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-07 13:20:32.306700: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-07 13:20:32.306713: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-07 13:20:32.306727: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-07 13:20:32.306740: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-07 13:20:32.308033: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-07 13:20:32.308066: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-07 13:20:32.308075: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-07 13:20:32.308083: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-07 13:20:32.309504: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9985 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2847 thcls : [{'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}] thcl {'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5275 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5275, 'learn_RUBBIA_REFUS_AMIENS_23', 16384, 25088, 'learn_RUBBIA_REFUS_AMIENS_23', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 23, 14, 19, 39), datetime.datetime(2021, 4, 23, 14, 19, 39)) {'thcl': {'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'], 'list_hashtags_csv': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'svm_hashtag_type_desc': 5275, 'photo_desc_type': 5275, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_RUBBIA_REFUS_AMIENS_23 NUM_CLASSES 9 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_RUBBIA_REFUS_AMIENS_23 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-07 13:20:41.356080: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-07 13:20:41.541272: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_RUBBIA_REFUS_AMIENS_23 /data/models_weight/learn_RUBBIA_REFUS_AMIENS_23/mask_model.h5 size_local : 256009536 size in s3 : 256009536 create time local : 2021-08-09 09:43:22 create time in s3 : 2021-08-06 18:54:04 mask_model.h5 already exist and didn't need to update list_images length : 2 NEW PHOTO pour l'instant on ne peut pas sauvegarder la photo dans les tile Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -113.62969 max: 145.24062 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 7 time for calcul the mask position with numpy : 0.000102996826171875 nb_pixel_total : 804 time to create 1 rle with old method : 0.0015380382537841797 length of segment : 27 time for calcul the mask position with numpy : 4.7206878662109375e-05 nb_pixel_total : 395 time to create 1 rle with old method : 0.0007052421569824219 length of segment : 37 time for calcul the mask position with numpy : 5.841255187988281e-05 nb_pixel_total : 15 time to create 1 rle with old method : 0.00010323524475097656 length of segment : 8 time for calcul the mask position with numpy : 6.771087646484375e-05 nb_pixel_total : 1179 time to create 1 rle with old method : 0.006267547607421875 length of segment : 66 time for calcul the mask position with numpy : 8.940696716308594e-05 nb_pixel_total : 5 time to create 1 rle with old method : 4.887580871582031e-05 length of segment : 4 time for calcul the mask position with numpy : 9.775161743164062e-05 nb_pixel_total : 941 time to create 1 rle with old method : 0.0018336772918701172 length of segment : 35 time for calcul the mask position with numpy : 0.0003695487976074219 nb_pixel_total : 7918 time to create 1 rle with old method : 0.01643061637878418 length of segment : 131 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -122.40703 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 3 time for calcul the mask position with numpy : 9.489059448242188e-05 nb_pixel_total : 2393 time to create 1 rle with old method : 0.0032639503479003906 length of segment : 51 time for calcul the mask position with numpy : 6.866455078125e-05 nb_pixel_total : 416 time to create 1 rle with old method : 0.0007190704345703125 length of segment : 49 time for calcul the mask position with numpy : 8.082389831542969e-05 nb_pixel_total : 243 time to create 1 rle with old method : 0.00041747093200683594 length of segment : 41 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -119.65703 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 5 time for calcul the mask position with numpy : 0.00014591217041015625 nb_pixel_total : 999 time to create 1 rle with old method : 0.002583026885986328 length of segment : 43 time for calcul the mask position with numpy : 0.0013685226440429688 nb_pixel_total : 2654 time to create 1 rle with old method : 0.013165950775146484 length of segment : 83 time for calcul the mask position with numpy : 5.8650970458984375e-05 nb_pixel_total : 444 time to create 1 rle with old method : 0.0007107257843017578 length of segment : 24 time for calcul the mask position with numpy : 7.510185241699219e-05 nb_pixel_total : 403 time to create 1 rle with old method : 0.0006406307220458984 length of segment : 24 length of segment : 0 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -105.97734 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 6 time for calcul the mask position with numpy : 4.744529724121094e-05 nb_pixel_total : 227 time to create 1 rle with old method : 0.0004248619079589844 length of segment : 16 time for calcul the mask position with numpy : 5.9604644775390625e-05 nb_pixel_total : 928 time to create 1 rle with old method : 0.0014081001281738281 length of segment : 63 time for calcul the mask position with numpy : 0.00024271011352539062 nb_pixel_total : 11011 time to create 1 rle with old method : 0.015252828598022461 length of segment : 91 time for calcul the mask position with numpy : 6.628036499023438e-05 nb_pixel_total : 299 time to create 1 rle with old method : 0.0005648136138916016 length of segment : 22 time for calcul the mask position with numpy : 3.9577484130859375e-05 nb_pixel_total : 195 time to create 1 rle with old method : 0.0002770423889160156 length of segment : 18 time for calcul the mask position with numpy : 4.38690185546875e-05 nb_pixel_total : 241 time to create 1 rle with old method : 0.0004630088806152344 length of segment : 22 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 141.50625 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 4 time for calcul the mask position with numpy : 4.00543212890625e-05 nb_pixel_total : 126 time to create 1 rle with old method : 0.00021767616271972656 length of segment : 10 time for calcul the mask position with numpy : 6.461143493652344e-05 nb_pixel_total : 1113 time to create 1 rle with old method : 0.0013432502746582031 length of segment : 113 time for calcul the mask position with numpy : 3.910064697265625e-05 nb_pixel_total : 506 time to create 1 rle with old method : 0.0007014274597167969 length of segment : 21 time for calcul the mask position with numpy : 4.458427429199219e-05 nb_pixel_total : 647 time to create 1 rle with old method : 0.0008392333984375 length of segment : 44 Processing 1 images image shape: (400, 400, 3) min: 8.00000 max: 232.00000 molded_images shape: (1, 640, 640, 3) min: -80.78984 max: 93.34453 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 1 time for calcul the mask position with numpy : 0.001314401626586914 nb_pixel_total : 154521 time to create 1 rle with new method : 0.0023698806762695312 length of segment : 399 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -104.70391 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 7 time for calcul the mask position with numpy : 8.416175842285156e-05 nb_pixel_total : 790 time to create 1 rle with old method : 0.0018584728240966797 length of segment : 18 time for calcul the mask position with numpy : 6.67572021484375e-05 nb_pixel_total : 1963 time to create 1 rle with old method : 0.002251148223876953 length of segment : 69 time for calcul the mask position with numpy : 0.00011730194091796875 nb_pixel_total : 3762 time to create 1 rle with old method : 0.0048978328704833984 length of segment : 102 time for calcul the mask position with numpy : 6.723403930664062e-05 nb_pixel_total : 463 time to create 1 rle with old method : 0.00086212158203125 length of segment : 26 time for calcul the mask position with numpy : 3.814697265625e-05 nb_pixel_total : 183 time to create 1 rle with old method : 0.00027751922607421875 length of segment : 13 time for calcul the mask position with numpy : 4.57763671875e-05 nb_pixel_total : 856 time to create 1 rle with old method : 0.0010759830474853516 length of segment : 31 time for calcul the mask position with numpy : 3.2901763916015625e-05 nb_pixel_total : 257 time to create 1 rle with old method : 0.00041985511779785156 length of segment : 13 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -114.22344 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 11 time for calcul the mask position with numpy : 4.601478576660156e-05 nb_pixel_total : 164 time to create 1 rle with old method : 0.00029206275939941406 length of segment : 17 time for calcul the mask position with numpy : 3.62396240234375e-05 nb_pixel_total : 139 time to create 1 rle with old method : 0.00026226043701171875 length of segment : 15 time for calcul the mask position with numpy : 3.266334533691406e-05 nb_pixel_total : 185 time to create 1 rle with old method : 0.00027251243591308594 length of segment : 18 time for calcul the mask position with numpy : 5.340576171875e-05 nb_pixel_total : 338 time to create 1 rle with old method : 0.0004968643188476562 length of segment : 78 time for calcul the mask position with numpy : 6.985664367675781e-05 nb_pixel_total : 1008 time to create 1 rle with old method : 0.0016546249389648438 length of segment : 47 time for calcul the mask position with numpy : 5.6743621826171875e-05 nb_pixel_total : 473 time to create 1 rle with old method : 0.0009725093841552734 length of segment : 19 time for calcul the mask position with numpy : 4.57763671875e-05 nb_pixel_total : 80 time to create 1 rle with old method : 0.0001735687255859375 length of segment : 9 time for calcul the mask position with numpy : 3.6716461181640625e-05 nb_pixel_total : 249 time to create 1 rle with old method : 0.00040841102600097656 length of segment : 21 time for calcul the mask position with numpy : 3.504753112792969e-05 nb_pixel_total : 283 time to create 1 rle with old method : 0.00044226646423339844 length of segment : 23 time for calcul the mask position with numpy : 3.790855407714844e-05 nb_pixel_total : 578 time to create 1 rle with old method : 0.00084686279296875 length of segment : 20 time for calcul the mask position with numpy : 3.9577484130859375e-05 nb_pixel_total : 659 time to create 1 rle with old method : 0.0008938312530517578 length of segment : 23 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -114.68438 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 15 time for calcul the mask position with numpy : 0.0002961158752441406 nb_pixel_total : 12082 time to create 1 rle with old method : 0.014782428741455078 length of segment : 237 time for calcul the mask position with numpy : 4.410743713378906e-05 nb_pixel_total : 181 time to create 1 rle with old method : 0.0002684593200683594 length of segment : 21 time for calcul the mask position with numpy : 9.846687316894531e-05 nb_pixel_total : 2060 time to create 1 rle with old method : 0.003057718276977539 length of segment : 68 time for calcul the mask position with numpy : 0.0002903938293457031 nb_pixel_total : 5730 time to create 1 rle with old method : 0.007935285568237305 length of segment : 167 time for calcul the mask position with numpy : 0.00011038780212402344 nb_pixel_total : 3533 time to create 1 rle with old method : 0.0046842098236083984 length of segment : 63 time for calcul the mask position with numpy : 4.601478576660156e-05 nb_pixel_total : 65 time to create 1 rle with old method : 0.00016736984252929688 length of segment : 9 time for calcul the mask position with numpy : 0.0002605915069580078 nb_pixel_total : 10674 time to create 1 rle with old method : 0.01425313949584961 length of segment : 211 time for calcul the mask position with numpy : 6.246566772460938e-05 nb_pixel_total : 150 time to create 1 rle with old method : 0.00028586387634277344 length of segment : 15 time for calcul the mask position with numpy : 4.506111145019531e-05 nb_pixel_total : 164 time to create 1 rle with old method : 0.0002751350402832031 length of segment : 24 time for calcul the mask position with numpy : 7.033348083496094e-05 nb_pixel_total : 882 time to create 1 rle with old method : 0.0012602806091308594 length of segment : 48 time for calcul the mask position with numpy : 3.981590270996094e-05 nb_pixel_total : 230 time to create 1 rle with old method : 0.0004067420959472656 length of segment : 21 time for calcul the mask position with numpy : 6.771087646484375e-05 nb_pixel_total : 1524 time to create 1 rle with old method : 0.0021941661834716797 length of segment : 73 time for calcul the mask position with numpy : 5.14984130859375e-05 nb_pixel_total : 272 time to create 1 rle with old method : 0.00043010711669921875 length of segment : 24 time for calcul the mask position with numpy : 0.0001633167266845703 nb_pixel_total : 572 time to create 1 rle with old method : 0.0011556148529052734 length of segment : 95 time for calcul the mask position with numpy : 4.982948303222656e-05 nb_pixel_total : 877 time to create 1 rle with old method : 0.0012433528900146484 length of segment : 37 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 145.38516 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 7 time for calcul the mask position with numpy : 5.53131103515625e-05 nb_pixel_total : 518 time to create 1 rle with old method : 0.000835418701171875 length of segment : 25 time for calcul the mask position with numpy : 4.649162292480469e-05 nb_pixel_total : 279 time to create 1 rle with old method : 0.00040221214294433594 length of segment : 47 time for calcul the mask position with numpy : 8.130073547363281e-05 nb_pixel_total : 546 time to create 1 rle with old method : 0.0011260509490966797 length of segment : 44 time for calcul the mask position with numpy : 8.702278137207031e-05 nb_pixel_total : 1552 time to create 1 rle with old method : 0.00212860107421875 length of segment : 80 time for calcul the mask position with numpy : 0.00024628639221191406 nb_pixel_total : 4203 time to create 1 rle with old method : 0.005812644958496094 length of segment : 218 time for calcul the mask position with numpy : 5.3882598876953125e-05 nb_pixel_total : 343 time to create 1 rle with old method : 0.0005054473876953125 length of segment : 27 time for calcul the mask position with numpy : 0.00019168853759765625 nb_pixel_total : 3454 time to create 1 rle with old method : 0.0044558048248291016 length of segment : 220 Processing 1 images image shape: (280, 400, 3) min: 19.00000 max: 222.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 90.59219 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 0 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 14 time for calcul the mask position with numpy : 5.4836273193359375e-05 nb_pixel_total : 390 time to create 1 rle with old method : 0.0006620883941650391 length of segment : 20 time for calcul the mask position with numpy : 4.4345855712890625e-05 nb_pixel_total : 640 time to create 1 rle with old method : 0.0010480880737304688 length of segment : 26 time for calcul the mask position with numpy : 4.839897155761719e-05 nb_pixel_total : 509 time to create 1 rle with old method : 0.0007977485656738281 length of segment : 34 time for calcul the mask position with numpy : 4.38690185546875e-05 nb_pixel_total : 349 time to create 1 rle with old method : 0.0006127357482910156 length of segment : 22 time for calcul the mask position with numpy : 6.4849853515625e-05 nb_pixel_total : 1626 time to create 1 rle with old method : 0.0022542476654052734 length of segment : 39 time for calcul the mask position with numpy : 5.650520324707031e-05 nb_pixel_total : 1073 time to create 1 rle with old method : 0.0015568733215332031 length of segment : 35 time for calcul the mask position with numpy : 4.4345855712890625e-05 nb_pixel_total : 338 time to create 1 rle with old method : 0.0005807876586914062 length of segment : 17 time for calcul the mask position with numpy : 3.814697265625e-05 nb_pixel_total : 109 time to create 1 rle with old method : 0.00020051002502441406 length of segment : 13 time for calcul the mask position with numpy : 4.1484832763671875e-05 nb_pixel_total : 251 time to create 1 rle with old method : 0.0004191398620605469 length of segment : 16 time for calcul the mask position with numpy : 7.390975952148438e-05 nb_pixel_total : 1813 time to create 1 rle with old method : 0.0025284290313720703 length of segment : 62 time for calcul the mask position with numpy : 7.081031799316406e-05 nb_pixel_total : 898 time to create 1 rle with old method : 0.0013413429260253906 length of segment : 35 time for calcul the mask position with numpy : 8.225440979003906e-05 nb_pixel_total : 2036 time to create 1 rle with old method : 0.0028405189514160156 length of segment : 95 time for calcul the mask position with numpy : 8.416175842285156e-05 nb_pixel_total : 1997 time to create 1 rle with old method : 0.002709627151489258 length of segment : 63 time for calcul the mask position with numpy : 5.650520324707031e-05 nb_pixel_total : 1580 time to create 1 rle with old method : 0.002417325973510742 length of segment : 38 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.06094 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 6 time for calcul the mask position with numpy : 5.078315734863281e-05 nb_pixel_total : 408 time to create 1 rle with old method : 0.0007688999176025391 length of segment : 21 time for calcul the mask position with numpy : 5.507469177246094e-05 nb_pixel_total : 70 time to create 1 rle with old method : 0.00014925003051757812 length of segment : 12 time for calcul the mask position with numpy : 3.7670135498046875e-05 nb_pixel_total : 376 time to create 1 rle with old method : 0.0006017684936523438 length of segment : 20 time for calcul the mask position with numpy : 5.364418029785156e-05 nb_pixel_total : 141 time to create 1 rle with old method : 0.00024700164794921875 length of segment : 15 time for calcul the mask position with numpy : 3.933906555175781e-05 nb_pixel_total : 254 time to create 1 rle with old method : 0.0004165172576904297 length of segment : 22 time for calcul the mask position with numpy : 7.43865966796875e-05 nb_pixel_total : 1479 time to create 1 rle with old method : 0.0021660327911376953 length of segment : 41 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 146.36172 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 14 time for calcul the mask position with numpy : 6.651878356933594e-05 nb_pixel_total : 1030 time to create 1 rle with old method : 0.0016357898712158203 length of segment : 27 time for calcul the mask position with numpy : 4.8160552978515625e-05 nb_pixel_total : 403 time to create 1 rle with old method : 0.0006442070007324219 length of segment : 38 time for calcul the mask position with numpy : 6.079673767089844e-05 nb_pixel_total : 104 time to create 1 rle with old method : 0.00021719932556152344 length of segment : 27 time for calcul the mask position with numpy : 0.00024008750915527344 nb_pixel_total : 11367 time to create 1 rle with old method : 0.015227317810058594 length of segment : 162 time for calcul the mask position with numpy : 5.53131103515625e-05 nb_pixel_total : 98 time to create 1 rle with old method : 0.000217437744140625 length of segment : 13 time for calcul the mask position with numpy : 0.00014257431030273438 nb_pixel_total : 536 time to create 1 rle with old method : 0.0007975101470947266 length of segment : 53 time for calcul the mask position with numpy : 7.62939453125e-05 nb_pixel_total : 227 time to create 1 rle with old method : 0.0004298686981201172 length of segment : 44 time for calcul the mask position with numpy : 3.886222839355469e-05 nb_pixel_total : 149 time to create 1 rle with old method : 0.0002512931823730469 length of segment : 18 time for calcul the mask position with numpy : 3.814697265625e-05 nb_pixel_total : 140 time to create 1 rle with old method : 0.0002484321594238281 length of segment : 15 time for calcul the mask position with numpy : 3.910064697265625e-05 nb_pixel_total : 382 time to create 1 rle with old method : 0.0006153583526611328 length of segment : 21 time for calcul the mask position with numpy : 4.7206878662109375e-05 nb_pixel_total : 376 time to create 1 rle with old method : 0.0006163120269775391 length of segment : 20 time for calcul the mask position with numpy : 8.654594421386719e-05 nb_pixel_total : 2233 time to create 1 rle with old method : 0.003304719924926758 length of segment : 65 time for calcul the mask position with numpy : 5.459785461425781e-05 nb_pixel_total : 291 time to create 1 rle with old method : 0.0005049705505371094 length of segment : 23 time for calcul the mask position with numpy : 0.0002837181091308594 nb_pixel_total : 11905 time to create 1 rle with old method : 0.015636444091796875 length of segment : 146 Processing 1 images image shape: (280, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 132.97500 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 4 time for calcul the mask position with numpy : 4.315376281738281e-05 nb_pixel_total : 83 time to create 1 rle with old method : 0.00014972686767578125 length of segment : 16 time for calcul the mask position with numpy : 3.409385681152344e-05 nb_pixel_total : 600 time to create 1 rle with old method : 0.0008184909820556641 length of segment : 18 time for calcul the mask position with numpy : 2.8848648071289062e-05 nb_pixel_total : 263 time to create 1 rle with old method : 0.00038552284240722656 length of segment : 11 time for calcul the mask position with numpy : 4.172325134277344e-05 nb_pixel_total : 425 time to create 1 rle with old method : 0.0005898475646972656 length of segment : 53 NEW PHOTO pour l'instant on ne peut pas sauvegarder la photo dans les tile Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -114.82891 max: 145.24062 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 5 time for calcul the mask position with numpy : 4.8160552978515625e-05 nb_pixel_total : 981 time to create 1 rle with old method : 0.0012488365173339844 length of segment : 37 time for calcul the mask position with numpy : 3.147125244140625e-05 nb_pixel_total : 324 time to create 1 rle with old method : 0.0003838539123535156 length of segment : 26 time for calcul the mask position with numpy : 3.0517578125e-05 nb_pixel_total : 4 time to create 1 rle with old method : 3.0994415283203125e-05 length of segment : 2 time for calcul the mask position with numpy : 2.9087066650390625e-05 nb_pixel_total : 45 time to create 1 rle with old method : 0.00011372566223144531 length of segment : 17 time for calcul the mask position with numpy : 2.9802322387695312e-05 nb_pixel_total : 65 time to create 1 rle with old method : 0.00011324882507324219 length of segment : 22 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -118.74297 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 3 time for calcul the mask position with numpy : 7.200241088867188e-05 nb_pixel_total : 2621 time to create 1 rle with old method : 0.0031249523162841797 length of segment : 55 time for calcul the mask position with numpy : 5.054473876953125e-05 nb_pixel_total : 356 time to create 1 rle with old method : 0.0004715919494628906 length of segment : 55 time for calcul the mask position with numpy : 3.7670135498046875e-05 nb_pixel_total : 161 time to create 1 rle with old method : 0.0002434253692626953 length of segment : 26 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -119.29375 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 5 time for calcul the mask position with numpy : 6.222724914550781e-05 nb_pixel_total : 992 time to create 1 rle with old method : 0.0015475749969482422 length of segment : 43 time for calcul the mask position with numpy : 5.078315734863281e-05 nb_pixel_total : 466 time to create 1 rle with old method : 0.0007655620574951172 length of segment : 22 time for calcul the mask position with numpy : 6.341934204101562e-05 nb_pixel_total : 226 time to create 1 rle with old method : 0.0006799697875976562 length of segment : 10 time for calcul the mask position with numpy : 9.179115295410156e-05 nb_pixel_total : 414 time to create 1 rle with old method : 0.0010447502136230469 length of segment : 27 time for calcul the mask position with numpy : 0.00012230873107910156 nb_pixel_total : 5539 time to create 1 rle with old method : 0.0065765380859375 length of segment : 94 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -109.88750 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 7 time for calcul the mask position with numpy : 4.363059997558594e-05 nb_pixel_total : 212 time to create 1 rle with old method : 0.000316619873046875 length of segment : 16 time for calcul the mask position with numpy : 5.0067901611328125e-05 nb_pixel_total : 1003 time to create 1 rle with old method : 0.001251220703125 length of segment : 54 time for calcul the mask position with numpy : 3.552436828613281e-05 nb_pixel_total : 261 time to create 1 rle with old method : 0.00038051605224609375 length of segment : 19 time for calcul the mask position with numpy : 0.0001888275146484375 nb_pixel_total : 3458 time to create 1 rle with old method : 0.004187583923339844 length of segment : 233 time for calcul the mask position with numpy : 0.00018024444580078125 nb_pixel_total : 9564 time to create 1 rle with old method : 0.011332511901855469 length of segment : 115 time for calcul the mask position with numpy : 4.1961669921875e-05 nb_pixel_total : 224 time to create 1 rle with old method : 0.0003113746643066406 length of segment : 21 time for calcul the mask position with numpy : 0.000213623046875 nb_pixel_total : 6344 time to create 1 rle with old method : 0.007444858551025391 length of segment : 127 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 145.08828 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 5 time for calcul the mask position with numpy : 4.458427429199219e-05 nb_pixel_total : 128 time to create 1 rle with old method : 0.00023174285888671875 length of segment : 11 time for calcul the mask position with numpy : 6.818771362304688e-05 nb_pixel_total : 3258 time to create 1 rle with old method : 0.003742694854736328 length of segment : 71 time for calcul the mask position with numpy : 3.1948089599609375e-05 nb_pixel_total : 202 time to create 1 rle with old method : 0.0002734661102294922 length of segment : 30 time for calcul the mask position with numpy : 6.29425048828125e-05 nb_pixel_total : 2659 time to create 1 rle with old method : 0.0031087398529052734 length of segment : 90 time for calcul the mask position with numpy : 4.9591064453125e-05 nb_pixel_total : 547 time to create 1 rle with old method : 0.0007445812225341797 length of segment : 22 Processing 1 images image shape: (400, 400, 3) min: 15.00000 max: 236.00000 molded_images shape: (1, 640, 640, 3) min: -76.45000 max: 97.41094 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 1 time for calcul the mask position with numpy : 0.0016167163848876953 nb_pixel_total : 153109 time to create 1 rle with new method : 0.0023202896118164062 length of segment : 395 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -105.41094 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 9 time for calcul the mask position with numpy : 8.678436279296875e-05 nb_pixel_total : 1398 time to create 1 rle with old method : 0.002702474594116211 length of segment : 56 time for calcul the mask position with numpy : 5.4836273193359375e-05 nb_pixel_total : 860 time to create 1 rle with old method : 0.00116729736328125 length of segment : 32 time for calcul the mask position with numpy : 4.8160552978515625e-05 nb_pixel_total : 1047 time to create 1 rle with old method : 0.001416921615600586 length of segment : 20 time for calcul the mask position with numpy : 0.00011730194091796875 nb_pixel_total : 6591 time to create 1 rle with old method : 0.007801055908203125 length of segment : 111 time for calcul the mask position with numpy : 3.600120544433594e-05 nb_pixel_total : 178 time to create 1 rle with old method : 0.0002567768096923828 length of segment : 13 time for calcul the mask position with numpy : 4.124641418457031e-05 nb_pixel_total : 776 time to create 1 rle with old method : 0.0009815692901611328 length of segment : 28 time for calcul the mask position with numpy : 3.933906555175781e-05 nb_pixel_total : 818 time to create 1 rle with old method : 0.001051187515258789 length of segment : 26 time for calcul the mask position with numpy : 3.24249267578125e-05 nb_pixel_total : 190 time to create 1 rle with old method : 0.0003161430358886719 length of segment : 11 time for calcul the mask position with numpy : 2.8133392333984375e-05 nb_pixel_total : 207 time to create 1 rle with old method : 0.000293731689453125 length of segment : 11 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -114.23516 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 8 time for calcul the mask position with numpy : 3.838539123535156e-05 nb_pixel_total : 143 time to create 1 rle with old method : 0.00026869773864746094 length of segment : 15 time for calcul the mask position with numpy : 4.410743713378906e-05 nb_pixel_total : 1125 time to create 1 rle with old method : 0.001392364501953125 length of segment : 51 time for calcul the mask position with numpy : 3.123283386230469e-05 nb_pixel_total : 232 time to create 1 rle with old method : 0.0003561973571777344 length of segment : 18 time for calcul the mask position with numpy : 3.981590270996094e-05 nb_pixel_total : 487 time to create 1 rle with old method : 0.0007452964782714844 length of segment : 18 time for calcul the mask position with numpy : 3.361701965332031e-05 nb_pixel_total : 312 time to create 1 rle with old method : 0.0004470348358154297 length of segment : 20 time for calcul the mask position with numpy : 4.0531158447265625e-05 nb_pixel_total : 717 time to create 1 rle with old method : 0.0009267330169677734 length of segment : 36 time for calcul the mask position with numpy : 3.0040740966796875e-05 nb_pixel_total : 203 time to create 1 rle with old method : 0.0002853870391845703 length of segment : 18 time for calcul the mask position with numpy : 3.552436828613281e-05 nb_pixel_total : 463 time to create 1 rle with old method : 0.0006539821624755859 length of segment : 21 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -109.71953 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 9 time for calcul the mask position with numpy : 0.00011730194091796875 nb_pixel_total : 5188 time to create 1 rle with old method : 0.006086826324462891 length of segment : 99 time for calcul the mask position with numpy : 0.00018596649169921875 nb_pixel_total : 8751 time to create 1 rle with old method : 0.009839773178100586 length of segment : 183 time for calcul the mask position with numpy : 3.743171691894531e-05 nb_pixel_total : 68 time to create 1 rle with old method : 0.0001251697540283203 length of segment : 9 time for calcul the mask position with numpy : 0.0001785755157470703 nb_pixel_total : 3972 time to create 1 rle with old method : 0.004628896713256836 length of segment : 96 time for calcul the mask position with numpy : 6.389617919921875e-05 nb_pixel_total : 1354 time to create 1 rle with old method : 0.0018048286437988281 length of segment : 49 time for calcul the mask position with numpy : 3.886222839355469e-05 nb_pixel_total : 775 time to create 1 rle with old method : 0.0008590221405029297 length of segment : 35 time for calcul the mask position with numpy : 3.5762786865234375e-05 nb_pixel_total : 390 time to create 1 rle with old method : 0.0005273818969726562 length of segment : 21 time for calcul the mask position with numpy : 3.361701965332031e-05 nb_pixel_total : 235 time to create 1 rle with old method : 0.0003542900085449219 length of segment : 19 time for calcul the mask position with numpy : 6.699562072753906e-05 nb_pixel_total : 1688 time to create 1 rle with old method : 0.0020437240600585938 length of segment : 76 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 150.01016 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 9 time for calcul the mask position with numpy : 4.100799560546875e-05 nb_pixel_total : 515 time to create 1 rle with old method : 0.0006568431854248047 length of segment : 24 time for calcul the mask position with numpy : 3.4809112548828125e-05 nb_pixel_total : 562 time to create 1 rle with old method : 0.0006735324859619141 length of segment : 43 time for calcul the mask position with numpy : 4.100799560546875e-05 nb_pixel_total : 200 time to create 1 rle with old method : 0.0002894401550292969 length of segment : 43 time for calcul the mask position with numpy : 6.4849853515625e-05 nb_pixel_total : 1714 time to create 1 rle with old method : 0.002044677734375 length of segment : 83 time for calcul the mask position with numpy : 7.677078247070312e-05 nb_pixel_total : 2696 time to create 1 rle with old method : 0.0033597946166992188 length of segment : 87 time for calcul the mask position with numpy : 3.790855407714844e-05 nb_pixel_total : 470 time to create 1 rle with old method : 0.0005738735198974609 length of segment : 28 time for calcul the mask position with numpy : 0.0002677440643310547 nb_pixel_total : 3796 time to create 1 rle with old method : 0.004340648651123047 length of segment : 235 time for calcul the mask position with numpy : 0.000194549560546875 nb_pixel_total : 2634 time to create 1 rle with old method : 0.0030264854431152344 length of segment : 200 time for calcul the mask position with numpy : 3.647804260253906e-05 nb_pixel_total : 426 time to create 1 rle with old method : 0.0005578994750976562 length of segment : 29 Processing 1 images image shape: (280, 400, 3) min: 15.00000 max: 223.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 92.98672 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 2 time for calcul the mask position with numpy : 6.985664367675781e-05 nb_pixel_total : 451 time to create 1 rle with old method : 0.0008189678192138672 length of segment : 53 time for calcul the mask position with numpy : 0.001024484634399414 nb_pixel_total : 106574 time to create 1 rle with old method : 0.11879539489746094 length of segment : 282 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 11 time for calcul the mask position with numpy : 4.38690185546875e-05 nb_pixel_total : 648 time to create 1 rle with old method : 0.0008454322814941406 length of segment : 26 time for calcul the mask position with numpy : 4.6253204345703125e-05 nb_pixel_total : 519 time to create 1 rle with old method : 0.0008866786956787109 length of segment : 32 time for calcul the mask position with numpy : 0.00010013580322265625 nb_pixel_total : 3936 time to create 1 rle with old method : 0.004484891891479492 length of segment : 118 time for calcul the mask position with numpy : 3.886222839355469e-05 nb_pixel_total : 431 time to create 1 rle with old method : 0.0005886554718017578 length of segment : 24 time for calcul the mask position with numpy : 4.0531158447265625e-05 nb_pixel_total : 572 time to create 1 rle with old method : 0.0007517337799072266 length of segment : 34 time for calcul the mask position with numpy : 3.528594970703125e-05 nb_pixel_total : 377 time to create 1 rle with old method : 0.0005278587341308594 length of segment : 23 time for calcul the mask position with numpy : 5.030632019042969e-05 nb_pixel_total : 1632 time to create 1 rle with old method : 0.0020723342895507812 length of segment : 39 time for calcul the mask position with numpy : 6.580352783203125e-05 nb_pixel_total : 2057 time to create 1 rle with old method : 0.0024271011352539062 length of segment : 63 time for calcul the mask position with numpy : 4.649162292480469e-05 nb_pixel_total : 973 time to create 1 rle with old method : 0.0012824535369873047 length of segment : 32 time for calcul the mask position with numpy : 5.555152893066406e-05 nb_pixel_total : 1590 time to create 1 rle with old method : 0.0020046234130859375 length of segment : 39 time for calcul the mask position with numpy : 3.0040740966796875e-05 nb_pixel_total : 101 time to create 1 rle with old method : 0.00016760826110839844 length of segment : 12 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 150.01016 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 7 time for calcul the mask position with numpy : 3.123283386230469e-05 nb_pixel_total : 60 time to create 1 rle with old method : 0.00011563301086425781 length of segment : 12 time for calcul the mask position with numpy : 3.337860107421875e-05 nb_pixel_total : 345 time to create 1 rle with old method : 0.0004611015319824219 length of segment : 19 time for calcul the mask position with numpy : 3.314018249511719e-05 nb_pixel_total : 283 time to create 1 rle with old method : 0.00037217140197753906 length of segment : 17 time for calcul the mask position with numpy : 3.0517578125e-05 nb_pixel_total : 187 time to create 1 rle with old method : 0.0002701282501220703 length of segment : 17 time for calcul the mask position with numpy : 2.86102294921875e-05 nb_pixel_total : 85 time to create 1 rle with old method : 0.00014019012451171875 length of segment : 11 time for calcul the mask position with numpy : 5.245208740234375e-05 nb_pixel_total : 2330 time to create 1 rle with old method : 0.0026509761810302734 length of segment : 43 time for calcul the mask position with numpy : 2.9325485229492188e-05 nb_pixel_total : 226 time to create 1 rle with old method : 0.00029778480529785156 length of segment : 19 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 145.24062 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 14 time for calcul the mask position with numpy : 4.744529724121094e-05 nb_pixel_total : 1036 time to create 1 rle with old method : 0.001142263412475586 length of segment : 48 time for calcul the mask position with numpy : 4.029273986816406e-05 nb_pixel_total : 75 time to create 1 rle with old method : 0.0001494884490966797 length of segment : 28 time for calcul the mask position with numpy : 3.790855407714844e-05 nb_pixel_total : 1029 time to create 1 rle with old method : 0.0011992454528808594 length of segment : 27 time for calcul the mask position with numpy : 3.123283386230469e-05 nb_pixel_total : 404 time to create 1 rle with old method : 0.0004906654357910156 length of segment : 21 time for calcul the mask position with numpy : 2.7418136596679688e-05 nb_pixel_total : 120 time to create 1 rle with old method : 0.00019359588623046875 length of segment : 15 time for calcul the mask position with numpy : 3.0517578125e-05 nb_pixel_total : 200 time to create 1 rle with old method : 0.00027060508728027344 length of segment : 23 time for calcul the mask position with numpy : 3.0040740966796875e-05 nb_pixel_total : 374 time to create 1 rle with old method : 0.00044226646423339844 length of segment : 25 time for calcul the mask position with numpy : 2.5272369384765625e-05 nb_pixel_total : 26 time to create 1 rle with old method : 5.7220458984375e-05 length of segment : 7 time for calcul the mask position with numpy : 0.00015878677368164062 nb_pixel_total : 11303 time to create 1 rle with old method : 0.011715888977050781 length of segment : 140 time for calcul the mask position with numpy : 3.0994415283203125e-05 nb_pixel_total : 138 time to create 1 rle with old method : 0.00019884109497070312 length of segment : 15 time for calcul the mask position with numpy : 5.269050598144531e-05 nb_pixel_total : 23 time to create 1 rle with old method : 4.792213439941406e-05 length of segment : 9 time for calcul the mask position with numpy : 2.8371810913085938e-05 nb_pixel_total : 52 time to create 1 rle with old method : 0.00011730194091796875 length of segment : 6 time for calcul the mask position with numpy : 5.602836608886719e-05 nb_pixel_total : 2127 time to create 1 rle with old method : 0.0023903846740722656 length of segment : 60 time for calcul the mask position with numpy : 2.956390380859375e-05 nb_pixel_total : 182 time to create 1 rle with old method : 0.00029587745666503906 length of segment : 14 Processing 1 images image shape: (280, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 133.03750 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 3 time for calcul the mask position with numpy : 3.4332275390625e-05 nb_pixel_total : 88 time to create 1 rle with old method : 0.0001556873321533203 length of segment : 16 time for calcul the mask position with numpy : 2.9087066650390625e-05 nb_pixel_total : 250 time to create 1 rle with old method : 0.0003821849822998047 length of segment : 12 time for calcul the mask position with numpy : 3.552436828613281e-05 nb_pixel_total : 597 time to create 1 rle with old method : 0.0008208751678466797 length of segment : 19 Detection mask done ! Trying to reset tf kernel 3748968 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5485 tf kernel not reseted sub process len(results) : 202 len(list_Values) 202 None max_time_sub_proc : 3600 parent process len(results) : 0 len(list_Values) 202 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 10774 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Catched exception ! Connect or reconnect ! Number saved : None batch 1 Loaded 202 chid ids of type : 4228 Number RLEs to save : 10098 save missing photos in datou_result : time spend for datou_step_exec : 25.44199848175049 time spend to save output : 1.124596357345581 total time spend for step 1 : 26.56659483909607 step2:brightness Fri Feb 7 13:20:54 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure inside step calcul brightness treat image : temp/1738930827_3748744_1335496257_c38c9d7e352a801efde9dfa1618767cf.jpg treat image : temp/1738930827_3748744_1335496255_546b8f5e406dddfc11b2de7cdf31161b.jpg Inside saveOutput : final : False verbose : 0 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 2 time used for this insertion : 0.013535261154174805 begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 2 time used for this insertion : 0.011481523513793945 save missing photos in datou_result : time spend for datou_step_exec : 0.4886167049407959 time spend to save output : 0.030056476593017578 total time spend for step 2 : 0.5186731815338135 step3:blur_detection Fri Feb 7 13:20:55 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure inside step blur_detection methode: ratio et variance treat image : temp/1738930827_3748744_1335496257_c38c9d7e352a801efde9dfa1618767cf.jpg resize: (1080, 1920) 1335496257 -8.060333889888044 treat image : temp/1738930827_3748744_1335496255_546b8f5e406dddfc11b2de7cdf31161b.jpg resize: (1080, 1920) 1335496255 -8.023597941276607 Inside saveOutput : final : False verbose : 0 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 2 time used for this insertion : 0.012636423110961914 begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 2 time used for this insertion : 0.012306451797485352 save missing photos in datou_result : time spend for datou_step_exec : 1.61238431930542 time spend to save output : 0.030785560607910156 total time spend for step 3 : 1.64316987991333 step4:rle_unique_nms_with_priority Fri Feb 7 13:20:56 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms batch 1 Loaded 202 chid ids of type : 4228 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++nb_obj : 103 nb_hashtags : 8 time to prepare the origin masks : 1.1607837677001953 time for calcul the mask position with numpy : 0.024076223373413086 nb_pixel_total : 1797493 time to create 1 rle with new method : 0.04861760139465332 time for calcul the mask position with numpy : 0.008443117141723633 nb_pixel_total : 1113 time to create 1 rle with old method : 0.0013470649719238281 time for calcul the mask position with numpy : 0.00850224494934082 nb_pixel_total : 2654 time to create 1 rle with old method : 0.0029637813568115234 time for calcul the mask position with numpy : 0.008486747741699219 nb_pixel_total : 243 time to create 1 rle with old method : 0.0003001689910888672 time for calcul the mask position with numpy : 0.008278608322143555 nb_pixel_total : 395 time to create 1 rle with old method : 0.00046515464782714844 time for calcul the mask position with numpy : 0.008124589920043945 nb_pixel_total : 150 time to create 1 rle with old method : 0.0002086162567138672 time for calcul the mask position with numpy : 0.008393049240112305 nb_pixel_total : 5 time to create 1 rle with old method : 1.811981201171875e-05 time for calcul the mask position with numpy : 0.008152484893798828 nb_pixel_total : 416 time to create 1 rle with old method : 0.000476837158203125 time for calcul the mask position with numpy : 0.008281230926513672 nb_pixel_total : 804 time to create 1 rle with old method : 0.0009381771087646484 time for calcul the mask position with numpy : 0.008377313613891602 nb_pixel_total : 1179 time to create 1 rle with old method : 0.0014042854309082031 time for calcul the mask position with numpy : 0.008217096328735352 nb_pixel_total : 15 time to create 1 rle with old method : 3.1948089599609375e-05 time for calcul the mask position with numpy : 0.008220434188842773 nb_pixel_total : 2393 time to create 1 rle with old method : 0.002645730972290039 time for calcul the mask position with numpy : 0.008103609085083008 nb_pixel_total : 999 time to create 1 rle with old method : 0.0010993480682373047 time for calcul the mask position with numpy : 0.008259057998657227 nb_pixel_total : 299 time to create 1 rle with old method : 0.000331878662109375 time for calcul the mask position with numpy : 0.008131980895996094 nb_pixel_total : 403 time to create 1 rle with old method : 0.00046825408935546875 time for calcul the mask position with numpy : 0.008133888244628906 nb_pixel_total : 834 time to create 1 rle with old method : 0.0009329319000244141 time for calcul the mask position with numpy : 0.008225679397583008 nb_pixel_total : 7918 time to create 1 rle with old method : 0.009120702743530273 time for calcul the mask position with numpy : 0.008311271667480469 nb_pixel_total : 227 time to create 1 rle with old method : 0.0002675056457519531 time for calcul the mask position with numpy : 0.008366823196411133 nb_pixel_total : 444 time to create 1 rle with old method : 0.0005021095275878906 time for calcul the mask position with numpy : 0.008117198944091797 nb_pixel_total : 195 time to create 1 rle with old method : 0.000232696533203125 time for calcul the mask position with numpy : 0.008377552032470703 nb_pixel_total : 241 time to create 1 rle with old method : 0.0002951622009277344 time for calcul the mask position with numpy : 0.0083770751953125 nb_pixel_total : 11011 time to create 1 rle with old method : 0.012436389923095703 time for calcul the mask position with numpy : 0.008449792861938477 nb_pixel_total : 126 time to create 1 rle with old method : 0.00015997886657714844 time for calcul the mask position with numpy : 0.008482933044433594 nb_pixel_total : 647 time to create 1 rle with old method : 0.0007596015930175781 time for calcul the mask position with numpy : 0.008392333984375 nb_pixel_total : 506 time to create 1 rle with old method : 0.0006039142608642578 time for calcul the mask position with numpy : 0.008881568908691406 nb_pixel_total : 154521 time to create 1 rle with new method : 0.03928661346435547 time for calcul the mask position with numpy : 0.008035421371459961 nb_pixel_total : 473 time to create 1 rle with old method : 0.0005872249603271484 time for calcul the mask position with numpy : 0.008247852325439453 nb_pixel_total : 5 time to create 1 rle with old method : 1.4066696166992188e-05 time for calcul the mask position with numpy : 0.00797128677368164 nb_pixel_total : 572 time to create 1 rle with old method : 0.0007030963897705078 time for calcul the mask position with numpy : 0.008199214935302734 nb_pixel_total : 877 time to create 1 rle with old method : 0.0010259151458740234 time for calcul the mask position with numpy : 0.008166074752807617 nb_pixel_total : 139 time to create 1 rle with old method : 0.00017380714416503906 time for calcul the mask position with numpy : 0.008265018463134766 nb_pixel_total : 5253 time to create 1 rle with old method : 0.0057332515716552734 time for calcul the mask position with numpy : 0.009862422943115234 nb_pixel_total : 3533 time to create 1 rle with old method : 0.0039899349212646484 time for calcul the mask position with numpy : 0.008099794387817383 nb_pixel_total : 2045 time to create 1 rle with old method : 0.0022847652435302734 time for calcul the mask position with numpy : 0.008312463760375977 nb_pixel_total : 546 time to create 1 rle with old method : 0.0006163120269775391 time for calcul the mask position with numpy : 0.008032798767089844 nb_pixel_total : 34 time to create 1 rle with old method : 7.295608520507812e-05 time for calcul the mask position with numpy : 0.00832366943359375 nb_pixel_total : 249 time to create 1 rle with old method : 0.00031948089599609375 time for calcul the mask position with numpy : 0.008501291275024414 nb_pixel_total : 164 time to create 1 rle with old method : 0.0002148151397705078 time for calcul the mask position with numpy : 0.012057781219482422 nb_pixel_total : 3762 time to create 1 rle with old method : 0.004361629486083984 time for calcul the mask position with numpy : 0.008395910263061523 nb_pixel_total : 182 time to create 1 rle with old method : 0.0002193450927734375 time for calcul the mask position with numpy : 0.008307695388793945 nb_pixel_total : 834 time to create 1 rle with old method : 0.0008893013000488281 time for calcul the mask position with numpy : 0.008132696151733398 nb_pixel_total : 12082 time to create 1 rle with old method : 0.012582778930664062 time for calcul the mask position with numpy : 0.008368492126464844 nb_pixel_total : 279 time to create 1 rle with old method : 0.00030875205993652344 time for calcul the mask position with numpy : 0.00809335708618164 nb_pixel_total : 4203 time to create 1 rle with old method : 0.004621267318725586 time for calcul the mask position with numpy : 0.008018732070922852 nb_pixel_total : 463 time to create 1 rle with old method : 0.0005657672882080078 time for calcul the mask position with numpy : 0.008089542388916016 nb_pixel_total : 148 time to create 1 rle with old method : 0.0002346038818359375 time for calcul the mask position with numpy : 0.008322954177856445 nb_pixel_total : 343 time to create 1 rle with old method : 0.0003731250762939453 time for calcul the mask position with numpy : 0.00815725326538086 nb_pixel_total : 10674 time to create 1 rle with old method : 0.011572837829589844 time for calcul the mask position with numpy : 0.008472919464111328 nb_pixel_total : 1963 time to create 1 rle with old method : 0.0023107528686523438 time for calcul the mask position with numpy : 0.008168458938598633 nb_pixel_total : 164 time to create 1 rle with old method : 0.00019931793212890625 time for calcul the mask position with numpy : 0.009444713592529297 nb_pixel_total : 89 time to create 1 rle with old method : 0.00014138221740722656 time for calcul the mask position with numpy : 0.008098602294921875 nb_pixel_total : 578 time to create 1 rle with old method : 0.0006785392761230469 time for calcul the mask position with numpy : 0.00808405876159668 nb_pixel_total : 701 time to create 1 rle with old method : 0.0008008480072021484 time for calcul the mask position with numpy : 0.008135080337524414 nb_pixel_total : 969 time to create 1 rle with old method : 0.0010864734649658203 time for calcul the mask position with numpy : 0.008017301559448242 nb_pixel_total : 181 time to create 1 rle with old method : 0.00020503997802734375 time for calcul the mask position with numpy : 0.007811546325683594 nb_pixel_total : 323 time to create 1 rle with old method : 0.0004191398620605469 time for calcul the mask position with numpy : 0.007953882217407227 nb_pixel_total : 42 time to create 1 rle with old method : 7.200241088867188e-05 time for calcul the mask position with numpy : 0.007799863815307617 nb_pixel_total : 230 time to create 1 rle with old method : 0.0002582073211669922 time for calcul the mask position with numpy : 0.0077495574951171875 nb_pixel_total : 150 time to create 1 rle with old method : 0.00016880035400390625 time for calcul the mask position with numpy : 0.007933616638183594 nb_pixel_total : 518 time to create 1 rle with old method : 0.0005419254302978516 time for calcul the mask position with numpy : 0.00808095932006836 nb_pixel_total : 185 time to create 1 rle with old method : 0.00021958351135253906 time for calcul the mask position with numpy : 0.008385658264160156 nb_pixel_total : 1552 time to create 1 rle with old method : 0.0017800331115722656 time for calcul the mask position with numpy : 0.008359432220458984 nb_pixel_total : 1008 time to create 1 rle with old method : 0.0011250972747802734 time for calcul the mask position with numpy : 0.008446455001831055 nb_pixel_total : 257 time to create 1 rle with old method : 0.00032401084899902344 time for calcul the mask position with numpy : 0.008351564407348633 nb_pixel_total : 790 time to create 1 rle with old method : 0.0008602142333984375 time for calcul the mask position with numpy : 0.008558034896850586 nb_pixel_total : 65 time to create 1 rle with old method : 0.00010156631469726562 time for calcul the mask position with numpy : 0.008548736572265625 nb_pixel_total : 338 time to create 1 rle with old method : 0.0004153251647949219 time for calcul the mask position with numpy : 0.008499383926391602 nb_pixel_total : 1626 time to create 1 rle with old method : 0.0019140243530273438 time for calcul the mask position with numpy : 0.00855565071105957 nb_pixel_total : 390 time to create 1 rle with old method : 0.0004801750183105469 time for calcul the mask position with numpy : 0.008552312850952148 nb_pixel_total : 536 time to create 1 rle with old method : 0.0006570816040039062 time for calcul the mask position with numpy : 0.008512496948242188 nb_pixel_total : 178 time to create 1 rle with old method : 0.000347137451171875 time for calcul the mask position with numpy : 0.008585214614868164 nb_pixel_total : 263 time to create 1 rle with old method : 0.00032591819763183594 time for calcul the mask position with numpy : 0.008601188659667969 nb_pixel_total : 13 time to create 1 rle with old method : 6.818771362304688e-05 time for calcul the mask position with numpy : 0.008611202239990234 nb_pixel_total : 230 time to create 1 rle with old method : 0.00030422210693359375 time for calcul the mask position with numpy : 0.009123802185058594 nb_pixel_total : 98 time to create 1 rle with old method : 0.00016570091247558594 time for calcul the mask position with numpy : 0.009498357772827148 nb_pixel_total : 1073 time to create 1 rle with old method : 0.001272439956665039 time for calcul the mask position with numpy : 0.008276700973510742 nb_pixel_total : 1030 time to create 1 rle with old method : 0.0011963844299316406 time for calcul the mask position with numpy : 0.008133649826049805 nb_pixel_total : 640 time to create 1 rle with old method : 0.0007460117340087891 time for calcul the mask position with numpy : 0.008300304412841797 nb_pixel_total : 509 time to create 1 rle with old method : 0.0006432533264160156 time for calcul the mask position with numpy : 0.008304595947265625 nb_pixel_total : 83 time to create 1 rle with old method : 9.441375732421875e-05 time for calcul the mask position with numpy : 0.008157968521118164 nb_pixel_total : 141 time to create 1 rle with old method : 0.00017452239990234375 time for calcul the mask position with numpy : 0.008157014846801758 nb_pixel_total : 349 time to create 1 rle with old method : 0.00040912628173828125 time for calcul the mask position with numpy : 0.00829935073852539 nb_pixel_total : 600 time to create 1 rle with old method : 0.0007271766662597656 time for calcul the mask position with numpy : 0.008542060852050781 nb_pixel_total : 188 time to create 1 rle with old method : 0.0002865791320800781 time for calcul the mask position with numpy : 0.008545160293579102 nb_pixel_total : 1813 time to create 1 rle with old method : 0.002154111862182617 time for calcul the mask position with numpy : 0.008205413818359375 nb_pixel_total : 104 time to create 1 rle with old method : 0.00014472007751464844 time for calcul the mask position with numpy : 0.007938861846923828 nb_pixel_total : 898 time to create 1 rle with old method : 0.001047372817993164 time for calcul the mask position with numpy : 0.007949113845825195 nb_pixel_total : 408 time to create 1 rle with old method : 0.00045180320739746094 time for calcul the mask position with numpy : 0.00801849365234375 nb_pixel_total : 109 time to create 1 rle with old method : 0.0001373291015625 time for calcul the mask position with numpy : 0.008337259292602539 nb_pixel_total : 11367 time to create 1 rle with old method : 0.012825489044189453 time for calcul the mask position with numpy : 0.008478164672851562 nb_pixel_total : 140 time to create 1 rle with old method : 0.00019931793212890625 time for calcul the mask position with numpy : 0.008238077163696289 nb_pixel_total : 821 time to create 1 rle with old method : 0.0010216236114501953 time for calcul the mask position with numpy : 0.008379936218261719 nb_pixel_total : 376 time to create 1 rle with old method : 0.0006535053253173828 time for calcul the mask position with numpy : 0.008188724517822266 nb_pixel_total : 403 time to create 1 rle with old method : 0.0004420280456542969 time for calcul the mask position with numpy : 0.008126974105834961 nb_pixel_total : 227 time to create 1 rle with old method : 0.0002918243408203125 time for calcul the mask position with numpy : 0.008226394653320312 nb_pixel_total : 382 time to create 1 rle with old method : 0.00044846534729003906 time for calcul the mask position with numpy : 0.008035421371459961 nb_pixel_total : 17 time to create 1 rle with old method : 3.6716461181640625e-05 time for calcul the mask position with numpy : 0.008054256439208984 nb_pixel_total : 2036 time to create 1 rle with old method : 0.002216339111328125 time for calcul the mask position with numpy : 0.008223295211791992 nb_pixel_total : 31 time to create 1 rle with old method : 8.392333984375e-05 time for calcul the mask position with numpy : 0.008138895034790039 nb_pixel_total : 2233 time to create 1 rle with old method : 0.002452850341796875 time for calcul the mask position with numpy : 0.008270740509033203 nb_pixel_total : 254 time to create 1 rle with old method : 0.0003101825714111328 time for calcul the mask position with numpy : 0.00838923454284668 nb_pixel_total : 70 time to create 1 rle with old method : 9.489059448242188e-05 time for calcul the mask position with numpy : 0.008332252502441406 nb_pixel_total : 291 time to create 1 rle with old method : 0.00034928321838378906 time for calcul the mask position with numpy : 0.008226633071899414 nb_pixel_total : 1479 time to create 1 rle with old method : 0.0016236305236816406 create new chi : 1.1205253601074219 after preparing all the mask , begin to delete the rle from the crop_hashtag_id => VR 28-11-20 : il faut déplacer cela apres le save_crop_hashtag_ids_obj de la ligne 9514 ! we have 0 chi objets contains the rles time to delete rle : 0.005719184875488281 batch 1 Loaded 104 chid ids of type : 4230 Number RLEs to save : 10494 TO DO : save crop sub photo not yet done ! save time : 0.6429486274719238 nb_obj : 97 nb_hashtags : 8 time to prepare the origin masks : 2.1429922580718994 time for calcul the mask position with numpy : 0.03116583824157715 nb_pixel_total : 1708162 time to create 1 rle with new method : 0.05962204933166504 time for calcul the mask position with numpy : 0.010739326477050781 nb_pixel_total : 161 time to create 1 rle with old method : 0.00019478797912597656 time for calcul the mask position with numpy : 0.009989023208618164 nb_pixel_total : 981 time to create 1 rle with old method : 0.0010726451873779297 time for calcul the mask position with numpy : 0.00985574722290039 nb_pixel_total : 45 time to create 1 rle with old method : 8.034706115722656e-05 time for calcul the mask position with numpy : 0.00986337661743164 nb_pixel_total : 65 time to create 1 rle with old method : 9.059906005859375e-05 time for calcul the mask position with numpy : 0.005726814270019531 nb_pixel_total : 324 time to create 1 rle with old method : 0.0003554821014404297 time for calcul the mask position with numpy : 0.005841970443725586 nb_pixel_total : 2621 time to create 1 rle with old method : 0.0027229785919189453 time for calcul the mask position with numpy : 0.0058116912841796875 nb_pixel_total : 4 time to create 1 rle with old method : 2.1457672119140625e-05 time for calcul the mask position with numpy : 0.00602269172668457 nb_pixel_total : 992 time to create 1 rle with old method : 0.001123189926147461 time for calcul the mask position with numpy : 0.006475210189819336 nb_pixel_total : 261 time to create 1 rle with old method : 0.0003268718719482422 time for calcul the mask position with numpy : 0.0062732696533203125 nb_pixel_total : 955 time to create 1 rle with old method : 0.0010838508605957031 time for calcul the mask position with numpy : 0.0058362483978271484 nb_pixel_total : 356 time to create 1 rle with old method : 0.0005176067352294922 time for calcul the mask position with numpy : 0.0057756900787353516 nb_pixel_total : 212 time to create 1 rle with old method : 0.0002651214599609375 time for calcul the mask position with numpy : 0.006000518798828125 nb_pixel_total : 202 time to create 1 rle with old method : 0.00024628639221191406 time for calcul the mask position with numpy : 0.010062932968139648 nb_pixel_total : 3458 time to create 1 rle with old method : 0.0037496089935302734 time for calcul the mask position with numpy : 0.009218454360961914 nb_pixel_total : 41 time to create 1 rle with old method : 7.295608520507812e-05 time for calcul the mask position with numpy : 0.005883216857910156 nb_pixel_total : 466 time to create 1 rle with old method : 0.0005280971527099609 time for calcul the mask position with numpy : 0.005921363830566406 nb_pixel_total : 226 time to create 1 rle with old method : 0.0002827644348144531 time for calcul the mask position with numpy : 0.005762577056884766 nb_pixel_total : 224 time to create 1 rle with old method : 0.00024819374084472656 time for calcul the mask position with numpy : 0.005782127380371094 nb_pixel_total : 5539 time to create 1 rle with old method : 0.006099224090576172 time for calcul the mask position with numpy : 0.005636453628540039 nb_pixel_total : 607 time to create 1 rle with old method : 0.0007507801055908203 time for calcul the mask position with numpy : 0.0058231353759765625 nb_pixel_total : 9564 time to create 1 rle with old method : 0.009841442108154297 time for calcul the mask position with numpy : 0.005853891372680664 nb_pixel_total : 128 time to create 1 rle with old method : 0.0002033710479736328 time for calcul the mask position with numpy : 0.005707979202270508 nb_pixel_total : 3258 time to create 1 rle with old method : 0.003686666488647461 time for calcul the mask position with numpy : 0.005642414093017578 nb_pixel_total : 18 time to create 1 rle with old method : 6.794929504394531e-05 time for calcul the mask position with numpy : 0.005888700485229492 nb_pixel_total : 547 time to create 1 rle with old method : 0.000629425048828125 time for calcul the mask position with numpy : 0.007302284240722656 nb_pixel_total : 153109 time to create 1 rle with new method : 0.026983261108398438 time for calcul the mask position with numpy : 0.005926609039306641 nb_pixel_total : 1156 time to create 1 rle with old method : 0.0013325214385986328 time for calcul the mask position with numpy : 0.005967617034912109 nb_pixel_total : 487 time to create 1 rle with old method : 0.0005640983581542969 time for calcul the mask position with numpy : 0.0061914920806884766 nb_pixel_total : 775 time to create 1 rle with old method : 0.0007951259613037109 time for calcul the mask position with numpy : 0.006165504455566406 nb_pixel_total : 5188 time to create 1 rle with old method : 0.005790233612060547 time for calcul the mask position with numpy : 0.006003141403198242 nb_pixel_total : 539 time to create 1 rle with old method : 0.0006558895111083984 time for calcul the mask position with numpy : 0.005914449691772461 nb_pixel_total : 143 time to create 1 rle with old method : 0.000179290771484375 time for calcul the mask position with numpy : 0.006029367446899414 nb_pixel_total : 562 time to create 1 rle with old method : 0.0006539821624755859 time for calcul the mask position with numpy : 0.005853176116943359 nb_pixel_total : 232 time to create 1 rle with old method : 0.0002949237823486328 time for calcul the mask position with numpy : 0.005939960479736328 nb_pixel_total : 5008 time to create 1 rle with old method : 0.00529789924621582 time for calcul the mask position with numpy : 0.005886077880859375 nb_pixel_total : 8751 time to create 1 rle with old method : 0.009580612182617188 time for calcul the mask position with numpy : 0.0058252811431884766 nb_pixel_total : 14 time to create 1 rle with old method : 3.838539123535156e-05 time for calcul the mask position with numpy : 0.0059206485748291016 nb_pixel_total : 860 time to create 1 rle with old method : 0.0009183883666992188 time for calcul the mask position with numpy : 0.005805253982543945 nb_pixel_total : 10 time to create 1 rle with old method : 3.123283386230469e-05 time for calcul the mask position with numpy : 0.005779743194580078 nb_pixel_total : 200 time to create 1 rle with old method : 0.0002357959747314453 time for calcul the mask position with numpy : 0.006025075912475586 nb_pixel_total : 3794 time to create 1 rle with old method : 0.004106998443603516 time for calcul the mask position with numpy : 0.006822347640991211 nb_pixel_total : 109 time to create 1 rle with old method : 0.00019431114196777344 time for calcul the mask position with numpy : 0.006024837493896484 nb_pixel_total : 470 time to create 1 rle with old method : 0.0005300045013427734 time for calcul the mask position with numpy : 0.006014585494995117 nb_pixel_total : 312 time to create 1 rle with old method : 0.0003743171691894531 time for calcul the mask position with numpy : 0.006246805191040039 nb_pixel_total : 1047 time to create 1 rle with old method : 0.001188039779663086 time for calcul the mask position with numpy : 0.007325172424316406 nb_pixel_total : 1398 time to create 1 rle with old method : 0.0016162395477294922 time for calcul the mask position with numpy : 0.00982666015625 nb_pixel_total : 2696 time to create 1 rle with old method : 0.003102540969848633 time for calcul the mask position with numpy : 0.010004043579101562 nb_pixel_total : 1688 time to create 1 rle with old method : 0.0017919540405273438 time for calcul the mask position with numpy : 0.009869813919067383 nb_pixel_total : 463 time to create 1 rle with old method : 0.0005705356597900391 time for calcul the mask position with numpy : 0.00976252555847168 nb_pixel_total : 390 time to create 1 rle with old method : 0.0004420280456542969 time for calcul the mask position with numpy : 0.009636640548706055 nb_pixel_total : 717 time to create 1 rle with old method : 0.0008361339569091797 time for calcul the mask position with numpy : 0.010802268981933594 nb_pixel_total : 515 time to create 1 rle with old method : 0.0006244182586669922 time for calcul the mask position with numpy : 0.00595402717590332 nb_pixel_total : 203 time to create 1 rle with old method : 0.000263214111328125 time for calcul the mask position with numpy : 0.005998849868774414 nb_pixel_total : 1714 time to create 1 rle with old method : 0.0019347667694091797 time for calcul the mask position with numpy : 0.006099700927734375 nb_pixel_total : 235 time to create 1 rle with old method : 0.00026988983154296875 time for calcul the mask position with numpy : 0.00575709342956543 nb_pixel_total : 1125 time to create 1 rle with old method : 0.0012905597686767578 time for calcul the mask position with numpy : 0.005823850631713867 nb_pixel_total : 25 time to create 1 rle with old method : 6.222724914550781e-05 time for calcul the mask position with numpy : 0.005899667739868164 nb_pixel_total : 190 time to create 1 rle with old method : 0.0002377033233642578 time for calcul the mask position with numpy : 0.005742073059082031 nb_pixel_total : 818 time to create 1 rle with old method : 0.0009295940399169922 time for calcul the mask position with numpy : 0.005816221237182617 nb_pixel_total : 68 time to create 1 rle with old method : 9.131431579589844e-05 time for calcul the mask position with numpy : 0.0063288211822509766 nb_pixel_total : 106123 time to create 1 rle with old method : 0.11167526245117188 time for calcul the mask position with numpy : 0.0059337615966796875 nb_pixel_total : 431 time to create 1 rle with old method : 0.0004699230194091797 time for calcul the mask position with numpy : 0.005991220474243164 nb_pixel_total : 1632 time to create 1 rle with old method : 0.0018889904022216797 time for calcul the mask position with numpy : 0.005872488021850586 nb_pixel_total : 519 time to create 1 rle with old method : 0.0005834102630615234 time for calcul the mask position with numpy : 0.005799531936645508 nb_pixel_total : 11 time to create 1 rle with old method : 4.506111145019531e-05 time for calcul the mask position with numpy : 0.005812406539916992 nb_pixel_total : 250 time to create 1 rle with old method : 0.00031828880310058594 time for calcul the mask position with numpy : 0.0057849884033203125 nb_pixel_total : 120 time to create 1 rle with old method : 0.0001461505889892578 time for calcul the mask position with numpy : 0.006090879440307617 nb_pixel_total : 973 time to create 1 rle with old method : 0.0010933876037597656 time for calcul the mask position with numpy : 0.00606989860534668 nb_pixel_total : 1029 time to create 1 rle with old method : 0.0012524127960205078 time for calcul the mask position with numpy : 0.005910634994506836 nb_pixel_total : 648 time to create 1 rle with old method : 0.0007452964782714844 time for calcul the mask position with numpy : 0.0058879852294921875 nb_pixel_total : 572 time to create 1 rle with old method : 0.0006504058837890625 time for calcul the mask position with numpy : 0.005717039108276367 nb_pixel_total : 187 time to create 1 rle with old method : 0.00020885467529296875 time for calcul the mask position with numpy : 0.005638837814331055 nb_pixel_total : 88 time to create 1 rle with old method : 0.00011229515075683594 time for calcul the mask position with numpy : 0.005725860595703125 nb_pixel_total : 2057 time to create 1 rle with old method : 0.0022187232971191406 time for calcul the mask position with numpy : 0.005715847015380859 nb_pixel_total : 377 time to create 1 rle with old method : 0.0004265308380126953 time for calcul the mask position with numpy : 0.005682468414306641 nb_pixel_total : 597 time to create 1 rle with old method : 0.0006387233734130859 time for calcul the mask position with numpy : 0.005733966827392578 nb_pixel_total : 283 time to create 1 rle with old method : 0.0003414154052734375 time for calcul the mask position with numpy : 0.0057125091552734375 nb_pixel_total : 75 time to create 1 rle with old method : 0.00011444091796875 time for calcul the mask position with numpy : 0.005642414093017578 nb_pixel_total : 11303 time to create 1 rle with old method : 0.011773109436035156 time for calcul the mask position with numpy : 0.00575566291809082 nb_pixel_total : 101 time to create 1 rle with old method : 0.0001308917999267578 time for calcul the mask position with numpy : 0.005923748016357422 nb_pixel_total : 138 time to create 1 rle with old method : 0.0001747608184814453 time for calcul the mask position with numpy : 0.005870342254638672 nb_pixel_total : 26 time to create 1 rle with old method : 4.9114227294921875e-05 time for calcul the mask position with numpy : 0.005936384201049805 nb_pixel_total : 52 time to create 1 rle with old method : 8.654594421386719e-05 time for calcul the mask position with numpy : 0.005944967269897461 nb_pixel_total : 345 time to create 1 rle with old method : 0.00041985511779785156 time for calcul the mask position with numpy : 0.005860090255737305 nb_pixel_total : 175 time to create 1 rle with old method : 0.00022411346435546875 time for calcul the mask position with numpy : 0.006043672561645508 nb_pixel_total : 200 time to create 1 rle with old method : 0.0002434253692626953 time for calcul the mask position with numpy : 0.005951404571533203 nb_pixel_total : 23 time to create 1 rle with old method : 4.792213439941406e-05 time for calcul the mask position with numpy : 0.006556510925292969 nb_pixel_total : 1036 time to create 1 rle with old method : 0.0014939308166503906 time for calcul the mask position with numpy : 0.006415843963623047 nb_pixel_total : 404 time to create 1 rle with old method : 0.000492095947265625 time for calcul the mask position with numpy : 0.005995035171508789 nb_pixel_total : 3936 time to create 1 rle with old method : 0.004502773284912109 time for calcul the mask position with numpy : 0.005907773971557617 nb_pixel_total : 451 time to create 1 rle with old method : 0.0005414485931396484 time for calcul the mask position with numpy : 0.005762815475463867 nb_pixel_total : 226 time to create 1 rle with old method : 0.0002484321594238281 time for calcul the mask position with numpy : 0.005837202072143555 nb_pixel_total : 2127 time to create 1 rle with old method : 0.002272367477416992 time for calcul the mask position with numpy : 0.005873441696166992 nb_pixel_total : 85 time to create 1 rle with old method : 0.00011849403381347656 time for calcul the mask position with numpy : 0.005988597869873047 nb_pixel_total : 60 time to create 1 rle with old method : 8.678436279296875e-05 time for calcul the mask position with numpy : 0.005992889404296875 nb_pixel_total : 2330 time to create 1 rle with old method : 0.002644777297973633 time for calcul the mask position with numpy : 0.00584864616394043 nb_pixel_total : 182 time to create 1 rle with old method : 0.0002415180206298828 create new chi : 0.9765145778656006 after preparing all the mask , begin to delete the rle from the crop_hashtag_id => VR 28-11-20 : il faut déplacer cela apres le save_crop_hashtag_ids_obj de la ligne 9514 ! we have 0 chi objets contains the rles time to delete rle : 0.0011708736419677734 batch 1 Loaded 98 chid ids of type : 4230 Number RLEs to save : 9804 TO DO : save crop sub photo not yet done ! save time : 0.7079083919525146 map_output_result : {1335496257: (0.0, 'Should be the crop_list due to order', 0.0), 1335496255: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1335496257, 1335496255] Looping around the photos to save general results len do output : 2 /1335496257.Didn't retrieve data . /1335496255.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2564355') ('4234', '20334353', '1335496257', None, None, None, None, None, '2564355') ('4234', None, None, None, None, None, None, None, '2564355') ('4234', '20334353', '1335496255', None, None, None, None, None, '2564355') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 6 time used for this insertion : 0.04289126396179199 save_final save missing photos in datou_result : time spend for datou_step_exec : 7.134265184402466 time spend to save output : 0.04313850402832031 total time spend for step 4 : 7.177403688430786 step5:crop_condition Fri Feb 7 13:21:03 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 4230 Loading chi in step crop for list_pids : 2 ! batch 1 Loaded 202 chid ids of type : 4230 begin to crop the class : papier param for this class : {'min_score': 0.6} filtre for class : papier hashtag_id of this class : 492668766 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 56 About to insert : list_path_to_insert length 56 new photo from crops ! About to upload 56 photos upload in portfolio : 4869462 init cache_photo without model_param we have 56 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738930865_3748744 we have uploaded 56 photos in the portfolio 4869462 time of upload the photos Elapsed time : 13.001113176345825 we have finished the crop for the class : papier begin to crop the class : carton param for this class : {'min_score': 0.6} filtre for class : carton hashtag_id of this class : 492774966 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 20 About to insert : list_path_to_insert length 20 new photo from crops ! About to upload 20 photos upload in portfolio : 4869462 init cache_photo without model_param we have 20 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738930878_3748744 we have uploaded 20 photos in the portfolio 4869462 time of upload the photos Elapsed time : 4.562381744384766 we have finished the crop for the class : carton begin to crop the class : metal param for this class : {'min_score': 0.6} filtre for class : metal hashtag_id of this class : 492628673 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738930883_3748744 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6376729011535645 we have finished the crop for the class : metal begin to crop the class : pet_clair param for this class : {'min_score': 0.6} filtre for class : pet_clair hashtag_id of this class : 2107755846 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 18 About to insert : list_path_to_insert length 18 new photo from crops ! About to upload 18 photos upload in portfolio : 4869462 init cache_photo without model_param we have 18 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738930884_3748744 we have uploaded 18 photos in the portfolio 4869462 time of upload the photos Elapsed time : 3.716189384460449 we have finished the crop for the class : pet_clair begin to crop the class : autre param for this class : {'min_score': 0.6} filtre for class : autre hashtag_id of this class : 494826614 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 29 About to insert : list_path_to_insert length 29 new photo from crops ! About to upload 29 photos upload in portfolio : 4869462 init cache_photo without model_param we have 29 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738930888_3748744 we have uploaded 29 photos in the portfolio 4869462 time of upload the photos Elapsed time : 8.007324695587158 we have finished the crop for the class : autre begin to crop the class : pehd param for this class : {'min_score': 0.6} filtre for class : pehd hashtag_id of this class : 628944319 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738930896_3748744 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.5432493686676025 we have finished the crop for the class : pehd begin to crop the class : pet_fonce param for this class : {'min_score': 0.6} filtre for class : pet_fonce hashtag_id of this class : 2107755900 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738930897_3748744 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6042358875274658 we have finished the crop for the class : pet_fonce delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1335496257, 1335496255] Looping around the photos to save general results len do output : 126 /1335514438Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514439Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514440Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514441Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514442Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514443Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514444Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514445Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514446Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514447Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514448Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514449Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514450Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514451Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514452Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514453Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514454Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514455Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514456Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514457Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514458Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514459Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514460Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514461Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514462Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514463Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514464Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514465Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514466Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514467Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514468Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514469Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514470Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514471Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514473Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514474Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514475Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514476Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514477Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514478Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514479Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514480Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514481Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514482Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514484Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514485Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514486Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514487Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514488Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514489Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514490Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514491Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514492Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514493Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514494Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514495Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514498Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514499Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514500Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514501Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514502Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514503Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514504Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514505Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514506Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514507Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514509Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514510Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514511Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514512Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514513Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514514Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514515Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514516Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514517Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514518Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514519Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514521Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514522Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514523Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514524Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514525Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514526Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514527Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514528Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514529Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514530Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514531Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514532Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514533Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514534Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514535Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514536Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514537Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514538Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514539Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514540Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514541Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514542Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514543Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514544Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514545Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514546Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514547Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514548Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514549Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514550Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514551Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514552Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514553Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514554Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514555Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514556Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514557Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514558Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514559Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514560Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514561Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514562Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514563Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514564Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514565Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514566Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514567Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514568Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335514569Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2564355') ('4234', '20334353', '1335496257', None, None, None, None, None, '2564355') ('4234', None, None, None, None, None, None, None, '2564355') ('4234', '20334353', '1335496255', None, None, None, None, None, '2564355') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 380 time used for this insertion : 0.02946758270263672 save_final save missing photos in datou_result : time spend for datou_step_exec : 34.796494483947754 time spend to save output : 0.03407430648803711 total time spend for step 5 : 34.83056879043579 step6:thcl Fri Feb 7 13:21:38 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! we are using the classfication for only one thcl 3237 time to import caffe and check if the image exist : 0.005674839019775391 time to convert the images to numpy array : 0.02704644203186035 time to import caffe and check if the image exist : 0.005074024200439453 time to convert the images to numpy array : 0.031350135803222656 time to import caffe and check if the image exist : 0.008077144622802734 time to convert the images to numpy array : 0.03324747085571289 time to import caffe and check if the image exist : 0.006852865219116211 time to convert the images to numpy array : 0.039481401443481445 time to import caffe and check if the image exist : 0.006928920745849609 time to convert the images to numpy array : 0.04218029975891113 time to import caffe and check if the image exist : 0.012634515762329102 time to convert the images to numpy array : 0.03834724426269531 time to import caffe and check if the image exist : 0.01585698127746582 time to convert the images to numpy array : 0.03701901435852051 time to import caffe and check if the image exist : 0.006904125213623047 time to convert the images to numpy array : 0.04529833793640137 time to import caffe and check if the image exist : 0.01700448989868164 time to convert the images to numpy array : 0.035253286361694336 time to import caffe and check if the image exist : 0.012553930282592773 time to convert the images to numpy array : 0.040891408920288086 total time to convert the images to numpy array : 0.2875404357910156 list photo_ids error: [] list photo_ids correct : [1335514561, 1335514562, 1335514563, 1335514564, 1335514565, 1335514566, 1335514567, 1335514568, 1335514569, 1335514464, 1335514465, 1335514466, 1335514467, 1335514468, 1335514469, 1335514470, 1335514471, 1335514473, 1335514474, 1335514475, 1335514476, 1335514477, 1335514438, 1335514439, 1335514440, 1335514441, 1335514442, 1335514443, 1335514444, 1335514445, 1335514446, 1335514447, 1335514448, 1335514449, 1335514450, 1335514492, 1335514493, 1335514494, 1335514495, 1335514498, 1335514499, 1335514500, 1335514501, 1335514502, 1335514503, 1335514504, 1335514505, 1335514506, 1335514535, 1335514536, 1335514537, 1335514538, 1335514539, 1335514540, 1335514541, 1335514542, 1335514543, 1335514544, 1335514545, 1335514546, 1335514547, 1335514478, 1335514479, 1335514480, 1335514481, 1335514482, 1335514484, 1335514485, 1335514486, 1335514487, 1335514488, 1335514489, 1335514490, 1335514491, 1335514451, 1335514452, 1335514453, 1335514454, 1335514455, 1335514456, 1335514457, 1335514458, 1335514459, 1335514460, 1335514461, 1335514462, 1335514463, 1335514522, 1335514523, 1335514524, 1335514525, 1335514526, 1335514527, 1335514528, 1335514529, 1335514530, 1335514531, 1335514532, 1335514533, 1335514534, 1335514548, 1335514549, 1335514550, 1335514551, 1335514552, 1335514553, 1335514554, 1335514555, 1335514556, 1335514557, 1335514558, 1335514559, 1335514560, 1335514507, 1335514509, 1335514510, 1335514511, 1335514512, 1335514513, 1335514514, 1335514515, 1335514516, 1335514517, 1335514518, 1335514519, 1335514521] number of photos to traite : 126 try to delete the photos incorrect in DB tagging for thcl : 3237 To do loadFromThcl(), then load ParamDescType : thcl3237 thcls : [{'id': 3237, 'mtr_user_id': 31, 'name': 'learn_rubbia_refus_2500', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton,Film_plastique,PEHD,PET_clair,PET_fonce,Papier,Tetrapak,flou,mal_croppe,metal,refus', 'svm_portfolios_learning': '4865689,4865690,4865686,4865684,4865685,4865688,4865691,4865693,4865692,4865687,4865683', 'photo_hashtag_type': 4158, 'photo_desc_type': 5561, 'type_classification': 'caffe', 'hashtag_id_list': '492774966,2107756122,628944319,2107755846,2107755900,492668766,609991870,492777938,2107755527,492628673,538914404'}] thcl {'id': 3237, 'mtr_user_id': 31, 'name': 'learn_rubbia_refus_2500', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton,Film_plastique,PEHD,PET_clair,PET_fonce,Papier,Tetrapak,flou,mal_croppe,metal,refus', 'svm_portfolios_learning': '4865689,4865690,4865686,4865684,4865685,4865688,4865691,4865693,4865692,4865687,4865683', 'photo_hashtag_type': 4158, 'photo_desc_type': 5561, 'type_classification': 'caffe', 'hashtag_id_list': '492774966,2107756122,628944319,2107755846,2107755900,492668766,609991870,492777938,2107755527,492628673,538914404'} Update svm_hashtag_type_desc : 5561 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5561, 'learn_rubbia_refus_2500', 2048, 2048, 'learn_rubbia_refus_2500', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2021, 12, 2, 19, 10, 8), datetime.datetime(2021, 12, 2, 19, 10, 8)) To loadFromThcl() : net_5561 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10774 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5561, 'learn_rubbia_refus_2500', 2048, 2048, 'learn_rubbia_refus_2500', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2021, 12, 2, 19, 10, 8), datetime.datetime(2021, 12, 2, 19, 10, 8)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_rubbia_refus_2500 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_rubbia_refus_2500 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_rubbia_refus_2500 /data/models_weight/learn_rubbia_refus_2500/caffemodel size_local : 94358479 size in s3 : 94358479 create time local : 2021-12-03 18:29:39 create time in s3 : 2021-12-02 17:49:16 caffemodel already exist and didn't need to update /data/models_weight/learn_rubbia_refus_2500/deploy.prototxt size_local : 32544 size in s3 : 32544 create time local : 2021-12-03 18:29:39 create time in s3 : 2021-12-02 17:49:15 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_rubbia_refus_2500/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-12-03 18:29:40 create time in s3 : 2021-12-02 18:09:52 mean.npy already exist and didn't need to update /data/models_weight/learn_rubbia_refus_2500/synset_words.txt size_local : 334 size in s3 : 334 create time local : 2021-12-03 18:29:40 create time in s3 : 2021-12-02 18:10:06 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_rubbia_refus_2500/deploy.prototxt caffemodel_filename : /data/models_weight/learn_rubbia_refus_2500/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10555 max_wait_temp : 1 max_wait : 0 dict_keys(['prob', 'pool5']) time used to do the prepocess of the images : 0.6157796382904053 time used to do the prediction : 0.531468391418457 save descriptor for thcl : 3237 time to traite the descriptors : 0.7287747859954834 Catched exception ! Connect or reconnect ! storage_type for insertDescriptorsMulti : 3 To insert : 1335514561 To insert : 1335514562 To insert : 1335514563 To insert : 1335514564 To insert : 1335514565 To insert : 1335514566 To insert : 1335514567 To insert : 1335514568 To insert : 1335514569 To insert : 1335514464 To insert : 1335514465 To insert : 1335514466 To insert : 1335514467 To insert : 1335514468 To insert : 1335514469 To insert : 1335514470 To insert : 1335514471 To insert : 1335514473 To insert : 1335514474 To insert : 1335514475 To insert : 1335514476 To insert : 1335514477 To insert : 1335514438 To insert : 1335514439 To insert : 1335514440 To insert : 1335514441 To insert : 1335514442 To insert : 1335514443 To insert : 1335514444 To insert : 1335514445 To insert : 1335514446 To insert : 1335514447 To insert : 1335514448 To insert : 1335514449 To insert : 1335514450 To insert : 1335514492 To insert : 1335514493 To insert : 1335514494 To insert : 1335514495 To insert : 1335514498 To insert : 1335514499 To insert : 1335514500 To insert : 1335514501 To insert : 1335514502 To insert : 1335514503 To insert : 1335514504 To insert : 1335514505 To insert : 1335514506 To insert : 1335514535 To insert : 1335514536 To insert : 1335514537 To insert : 1335514538 To insert : 1335514539 To insert : 1335514540 To insert : 1335514541 To insert : 1335514542 To insert : 1335514543 To insert : 1335514544 To insert : 1335514545 To insert : 1335514546 To insert : 1335514547 To insert : 1335514478 To insert : 1335514479 To insert : 1335514480 To insert : 1335514481 To insert : 1335514482 To insert : 1335514484 To insert : 1335514485 To insert : 1335514486 To insert : 1335514487 To insert : 1335514488 To insert : 1335514489 To insert : 1335514490 To insert : 1335514491 To insert : 1335514451 To insert : 1335514452 To insert : 1335514453 To insert : 1335514454 To insert : 1335514455 To insert : 1335514456 To insert : 1335514457 To insert : 1335514458 To insert : 1335514459 To insert : 1335514460 To insert : 1335514461 To insert : 1335514462 To insert : 1335514463 To insert : 1335514522 To insert : 1335514523 To insert : 1335514524 To insert : 1335514525 To insert : 1335514526 To insert : 1335514527 To insert : 1335514528 To insert : 1335514529 To insert : 1335514530 To insert : 1335514531 To insert : 1335514532 To insert : 1335514533 To insert : 1335514534 To insert : 1335514548 To insert : 1335514549 To insert : 1335514550 To insert : 1335514551 To insert : 1335514552 To insert : 1335514553 To insert : 1335514554 To insert : 1335514555 To insert : 1335514556 To insert : 1335514557 To insert : 1335514558 To insert : 1335514559 To insert : 1335514560 To insert : 1335514507 To insert : 1335514509 To insert : 1335514510 To insert : 1335514511 To insert : 1335514512 To insert : 1335514513 To insert : 1335514514 To insert : 1335514515 To insert : 1335514516 To insert : 1335514517 To insert : 1335514518 To insert : 1335514519 To insert : 1335514521 time to insert the descriptors : 21.041528701782227 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos SAVE THCL : begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 195 time used for this insertion : 0.019713640213012695 save missing photos in datou_result : time spend for datou_step_exec : 27.176450729370117 time spend to save output : 0.06602358818054199 total time spend for step 6 : 27.24247431755066 step7:ventilate_hashtags_in_portfolio Fri Feb 7 13:22:06 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! Iterating over portfolio : 20334353 get user id for portfolio 20334353 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20334353 AND mptpi.`type`=4230 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('refus','Carton','PET_clair','PET_fonce','autre','pehd','environnement','pet_clair','pet_fonce','carton','flou','papier','mal_croppe','PEHD','Tetrapak','Papier','metal','Film_plastique')) AND mptpi.`min_score`=0.6 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20334353 AND mptpi.`type`=4230 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('refus','Carton','PET_clair','PET_fonce','autre','pehd','environnement','pet_clair','pet_fonce','carton','flou','papier','mal_croppe','PEHD','Tetrapak','Papier','metal','Film_plastique')) AND mptpi.`min_score`=0.6 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20334353 AND mptpi.`type`=4231 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('refus','Carton','PET_clair','PET_fonce','autre','pehd','environnement','pet_clair','pet_fonce','carton','flou','papier','mal_croppe','PEHD','Tetrapak','Papier','metal','Film_plastique')) AND mptpi.`min_score`=0.6 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20334353 AND mptpi.`type`=4231 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('refus','Carton','PET_clair','PET_fonce','autre','pehd','environnement','pet_clair','pet_fonce','carton','flou','papier','mal_croppe','PEHD','Tetrapak','Papier','metal','Film_plastique')) AND mptpi.`min_score`=0.6 To do lien utilise dans velours : https://www.fotonower.com/velours/20334449,20334458,20334456,20334457,20334453,20334462,20334455,20334459,20334464,20334461,20334463,20334465,20334466?tags=refus,carton,pet_clair,pet_fonce,autre,pehd,environnement,flou,papier,mal_croppe,tetrapak,metal,film_plastique&datou_id_consolidate=4235&port_consolidate=20334353 Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1335496257, 1335496255] Looping around the photos to save general results len do output : 1 /20334353. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2564355') ('4234', '20334353', '1335496257', None, None, None, None, None, '2564355') ('4234', None, None, None, None, None, None, None, '2564355') ('4234', '20334353', '1335496255', None, None, None, None, None, '2564355') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 time used for this insertion : 0.012150287628173828 save_final save missing photos in datou_result : time spend for datou_step_exec : 2.3092994689941406 time spend to save output : 0.012408018112182617 total time spend for step 7 : 2.3217074871063232 step8:final Fri Feb 7 13:22:08 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1335496257: ('0.13539523799071332',), 1335496255: ('0.13539523799071332',)} new output for save of step final : {1335496257: ('0.13539523799071332',), 1335496255: ('0.13539523799071332',)} [1335496257, 1335496255] Looping around the photos to save general results len do output : 2 /1335496257.Didn't retrieve data . /1335496255.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2564355') ('4234', '20334353', '1335496257', None, None, None, None, None, '2564355') ('4234', None, None, None, None, None, None, None, '2564355') ('4234', '20334353', '1335496255', None, None, None, None, None, '2564355') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 6 time used for this insertion : 0.012503623962402344 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.11776852607727051 time spend to save output : 0.01273202896118164 total time spend for step 8 : 0.13050055503845215 step9:velours_tree Fri Feb 7 13:22:08 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure list_portfolios : 20334449,20334458,20334456,20334457,20334453,20334462,20334455,20334459,20334464,20334461,20334463,20334465,20334466 photo desc type : 5561 - Retrieving photos to tag... query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20334449 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20334458 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20334456 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20334457 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20334453 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20334462 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20334455 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20334459 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20334464 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20334461 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20334463 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20334465 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20334466 ORDER BY ph.size desc - Loading descriptors... Size : 2048 len(descriptors) : 126 Compute structured hierarchical clustering... ward : AgglomerativeClustering(n_clusters=6) ward.labels_ : [2 2 2 0 2 4 1 1 0 0 2 0 1 4 1 1 4 1 1 1 1 0 2 0 0 0 0 2 3 1 2 2 0 0 2 1 1 1 0 0 0 1 1 4 0 1 4 1 1 2 0 1 0 0 3 1 5 1 5 0 0 0 2 3 0 5 1 0 2 0 0 2 3 3 0 0 1 5 3 1 1 0 3 3 3 2 4 2 1 3 0 1 3 3 2 5 2 0 4 4 1 4 1 1 1 5 1 1 1 5 0 2 2 4 1 4 1 3 1 0 1 1 1 1 0 0] Elapsed time: 0.021599292755126953 graph_id used : 76737 - Beta version, working pretty good on 11-5-16 ! https://www.fotonower.com/velours/20334449,20334458,20334456,20334457,20334453,20334462,20334455,20334459,20334464,20334461,20334463,20334465,20334466?tags=refus,carton,pet_clair,pet_fonce,autre,pehd,environnement,flou,papier,mal_croppe,tetrapak,metal,film_plastique&datou_id_consolidate=4235&port_consolidate=20334353&tree_id=76737 Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : velours_tree we use saveGeneral [1335496257, 1335496255] Looping around the photos to save general results len do output : 1 /20334353Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2564355') ('4234', '20334353', '1335496257', None, None, None, None, None, '2564355') ('4234', None, None, None, None, None, None, None, '2564355') ('4234', '20334353', '1335496255', None, None, None, None, None, '2564355') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 time used for this insertion : 0.011872291564941406 save_final save missing photos in datou_result : time spend for datou_step_exec : 8.767855167388916 time spend to save output : 0.012064456939697266 total time spend for step 9 : 8.779919624328613 step10:send_mail_cod Fri Feb 7 13:22:17 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Qualipapia_P20334353_07-02-2025_13_22_17.pdf 20334431 imagette203344311738930937 20334440 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203344401738930937 20334438 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203344381738930938 20334439 change filename to text .imagette203344391738930939 20334435 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203344351738930939 20334444 change filename to text .imagette203344441738930940 20334441 imagette203344411738930940 20334446 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203344461738930940 20334443 imagette203344431738930941 20334445 imagette203344451738930941 20334447 change filename to text .imagette203344471738930941 20334448 imagette203344481738930941 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20334353 and hashtag_type = 4230 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20334449,20334458,20334456,20334457,20334453,20334462,20334455,20334459,20334464,20334461,20334463,20334465,20334466?tags=refus,carton,pet_clair,pet_fonce,autre,pehd,environnement,flou,papier,mal_croppe,tetrapak,metal,film_plastique&datou_id_consolidate=4235&port_consolidate=20334353&tree_id=76737 your option no_mail is active, we will not send the real mail to your client args[1335496257] : ((1335496257, -8.060333889888044, 492609224), (1335496257, -0.2932847445273235, 496442774), '0.13539523799071332') apple ((1335496257, -8.060333889888044, 492609224), (1335496257, -0.2932847445273235, 496442774), '0.13539523799071332') We are sending mail with results at cod@fotonower.com args[1335496255] : ((1335496255, -8.023597941276607, 492609224), (1335496255, -0.294271386968752, 496442774), '0.13539523799071332') apple ((1335496255, -8.023597941276607, 492609224), (1335496255, -0.294271386968752, 496442774), '0.13539523799071332') We are sending mail with results at cod@fotonower.com refus_total : 0.13539523799071332 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20334353 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20334353_07-02-2025_13_22_17.pdf results_Qualipapia_P20334353_07-02-2025_13_22_17.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20334353_07-02-2025_13_22_17.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('4234','20334353','results_Qualipapia_P20334353_07-02-2025_13_22_17.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20334353_07-02-2025_13_22_17.pdf','pdf','','0.17','0.13539523799071332') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1335496257, 1335496255] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2564355') ('4234', '20334353', '1335496257', None, None, None, None, None, '2564355') ('4234', None, None, None, None, None, None, None, '2564355') ('4234', '20334353', '1335496255', None, None, None, None, None, '2564355') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 2 time used for this insertion : 0.013459920883178711 save_final save missing photos in datou_result : time spend for datou_step_exec : 4.8002612590789795 time spend to save output : 0.01359701156616211 total time spend for step 10 : 4.813858270645142 step11:split_time_score Fri Feb 7 13:22:22 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 3442, 'mtr_user_id': 31, 'name': 'classifieur_2camions_valcor_021122_v1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'deux_camions,camion_droite,camion_gauche,pas_de_camion', 'svm_portfolios_learning': '7659379,7659034,7657685,7657114', 'photo_hashtag_type': 4458, 'photo_desc_type': 5723, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107760533,2107760534,2107760535,2107760536'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('09', 2),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 07022025 20334353 Nombre de photos uploadées : 2 / 23040 (0%) 07022025 20334353 Nombre de photos taguées (types de déchets): 2 / 2 (100%) 07022025 20334353 Nombre de photos taguées (volume) : 0 / 2 (0%) elapsed_time : load_data_split_time_score 1.1920928955078125e-06 elapsed_time : order_list_meta_photo_and_scores 5.4836273193359375e-06 ?? elapsed_time : fill_and_build_computed_from_old_data 0.000202178955078125 elapsed_time : insert_dashboard_record_day_entry 0.026232242584228516 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20333287 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20333288 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20333289 order by id desc limit 1 Qualite : 0.1470242564007996 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20333314_07-02-2025_12_23_00.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20333314 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20333314 AND mptpi.`type`=4230 To do Qualite : 0.18915574965371404 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20333315_07-02-2025_12_34_00.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20333315 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20333315 AND mptpi.`type`=4230 To do Qualite : 0.13539523799071332 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20334353_07-02-2025_13_22_17.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20334353 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20334353 AND mptpi.`type`=4230 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20334354 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20333794 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20333795 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20333796 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20333797 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20333798 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20333799 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20333800 order by id desc limit 1 Qualite : 0.2358425751212987 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20333801_07-02-2025_12_50_25.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20333801 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20333801 AND mptpi.`type`=4230 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'07022025': {'nb_upload': 2, 'nb_taggue_class': 2, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1335496257, 1335496255] Looping around the photos to save general results len do output : 1 /20334353Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2564355') ('4234', '20334353', '1335496257', None, None, None, None, None, '2564355') ('4234', None, None, None, None, None, None, None, '2564355') ('4234', '20334353', '1335496255', None, None, None, None, None, '2564355') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 time used for this insertion : 0.01211857795715332 save_final save missing photos in datou_result : time spend for datou_step_exec : 8.741158723831177 time spend to save output : 0.012265205383300781 total time spend for step 11 : 8.753423929214478 caffe_path_current : About to save ! 2 After save, about to update current ! ret : 1 len(input) + len(total_photo_id_missing) : 2 set_done_treatment 32.38user 21.45system 2:05.25elapsed 42%CPU (0avgtext+0avgdata 2499060maxresident)k 1034608inputs+11464outputs (1739major+1944359minor)pagefaults 0swaps