python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 4234 -l 5 ' -s datou_current_4234 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3813386 load datou : 4234 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5275, 'learn_RUBBIA_REFUS_AMIENS_23', 16384, 25088, 'learn_RUBBIA_REFUS_AMIENS_23', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 23, 14, 19, 39), datetime.datetime(2021, 4, 23, 14, 19, 39)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5561, 'learn_rubbia_refus_2500', 2048, 2048, 'learn_rubbia_refus_2500', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2021, 12, 2, 19, 10, 8), datetime.datetime(2021, 12, 2, 19, 10, 8)) load thcls load THCL from format json or kwargs add thcl : 2847 in CacheModelConfig load THCL from format json or kwargs add thcl : 3237 in CacheModelConfig load pdts add pdt : 5275 in CacheModelConfig add pdt : 5561 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 list_input_json: [] Current got : datou_id : 4234, datou_cur_ids : ['2627707'] with mtr_portfolio_ids : ['20957646'] and first list_photo_ids : [] new path : /proc/3813386/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, brightness, blur_detection, rle_unique_nms_with_priority, crop_condition, thcl, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , BFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 2 ; length of list_pids : 2 ; length of list_args : 2 time to download the photos : 0.3045215606689453 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Mon Mar 3 01:20:32 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10814 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-03-03 01:20:34.925042: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-03 01:20:34.951057: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-03 01:20:34.953156: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f4a68000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-03 01:20:34.953214: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-03 01:20:34.957299: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-03 01:20:35.201405: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x37474cf0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-03 01:20:35.201454: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-03 01:20:35.202426: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-03 01:20:35.202755: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-03 01:20:35.204695: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-03 01:20:35.206680: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-03 01:20:35.207019: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-03 01:20:35.209104: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-03 01:20:35.210112: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-03 01:20:35.214257: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-03 01:20:35.215717: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-03 01:20:35.215789: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-03 01:20:35.216524: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-03 01:20:35.216539: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-03 01:20:35.216548: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-03 01:20:35.221349: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10023 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-03-03 01:20:35.517415: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-03 01:20:35.517527: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-03 01:20:35.517543: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-03 01:20:35.517558: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-03 01:20:35.517572: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-03 01:20:35.517586: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-03 01:20:35.517599: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-03 01:20:35.517613: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-03 01:20:35.518776: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-03 01:20:35.519942: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-03 01:20:35.519972: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-03 01:20:35.519987: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-03 01:20:35.520001: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-03 01:20:35.520015: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-03 01:20:35.520029: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-03 01:20:35.520043: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-03 01:20:35.520057: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-03 01:20:35.521215: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-03 01:20:35.521248: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-03 01:20:35.521256: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-03 01:20:35.521263: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-03 01:20:35.522450: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10023 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2847 thcls : [{'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}] thcl {'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5275 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5275, 'learn_RUBBIA_REFUS_AMIENS_23', 16384, 25088, 'learn_RUBBIA_REFUS_AMIENS_23', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 23, 14, 19, 39), datetime.datetime(2021, 4, 23, 14, 19, 39)) {'thcl': {'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'], 'list_hashtags_csv': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'svm_hashtag_type_desc': 5275, 'photo_desc_type': 5275, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_RUBBIA_REFUS_AMIENS_23 NUM_CLASSES 9 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_RUBBIA_REFUS_AMIENS_23 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-03-03 01:20:42.386213: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-03 01:20:42.582748: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_RUBBIA_REFUS_AMIENS_23 /data/models_weight/learn_RUBBIA_REFUS_AMIENS_23/mask_model.h5 size_local : 256009536 size in s3 : 256009536 create time local : 2021-08-09 09:43:22 create time in s3 : 2021-08-06 18:54:04 mask_model.h5 already exist and didn't need to update list_images length : 2 NEW PHOTO pour l'instant on ne peut pas sauvegarder la photo dans les tile Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -119.24688 max: 150.01016 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 9 time for calcul the mask position with numpy : 7.843971252441406e-05 nb_pixel_total : 623 time to create 1 rle with old method : 0.0012884140014648438 length of segment : 54 time for calcul the mask position with numpy : 8.797645568847656e-05 nb_pixel_total : 462 time to create 1 rle with old method : 0.0011744499206542969 length of segment : 25 time for calcul the mask position with numpy : 7.82012939453125e-05 nb_pixel_total : 660 time to create 1 rle with old method : 0.001641988754272461 length of segment : 61 time for calcul the mask position with numpy : 7.43865966796875e-05 nb_pixel_total : 1156 time to create 1 rle with old method : 0.002180337905883789 length of segment : 35 time for calcul the mask position with numpy : 6.532669067382812e-05 nb_pixel_total : 909 time to create 1 rle with old method : 0.0017414093017578125 length of segment : 38 time for calcul the mask position with numpy : 5.316734313964844e-05 nb_pixel_total : 940 time to create 1 rle with old method : 0.001779794692993164 length of segment : 41 time for calcul the mask position with numpy : 4.076957702636719e-05 nb_pixel_total : 198 time to create 1 rle with old method : 0.0004191398620605469 length of segment : 12 time for calcul the mask position with numpy : 5.0067901611328125e-05 nb_pixel_total : 1133 time to create 1 rle with old method : 0.001588582992553711 length of segment : 26 time for calcul the mask position with numpy : 3.981590270996094e-05 nb_pixel_total : 828 time to create 1 rle with old method : 0.0011620521545410156 length of segment : 41 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -117.21172 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 12 time for calcul the mask position with numpy : 4.363059997558594e-05 nb_pixel_total : 193 time to create 1 rle with old method : 0.0003120899200439453 length of segment : 23 time for calcul the mask position with numpy : 6.031990051269531e-05 nb_pixel_total : 2506 time to create 1 rle with old method : 0.0033111572265625 length of segment : 46 time for calcul the mask position with numpy : 3.0994415283203125e-05 nb_pixel_total : 98 time to create 1 rle with old method : 0.00016736984252929688 length of segment : 12 time for calcul the mask position with numpy : 8.082389831542969e-05 nb_pixel_total : 3509 time to create 1 rle with old method : 0.0048334598541259766 length of segment : 46 time for calcul the mask position with numpy : 0.00016951560974121094 nb_pixel_total : 875 time to create 1 rle with old method : 0.0012848377227783203 length of segment : 91 time for calcul the mask position with numpy : 3.218650817871094e-05 nb_pixel_total : 166 time to create 1 rle with old method : 0.0002727508544921875 length of segment : 13 time for calcul the mask position with numpy : 4.1484832763671875e-05 nb_pixel_total : 866 time to create 1 rle with old method : 0.0011568069458007812 length of segment : 36 time for calcul the mask position with numpy : 2.7894973754882812e-05 nb_pixel_total : 29 time to create 1 rle with old method : 6.723403930664062e-05 length of segment : 5 time for calcul the mask position with numpy : 4.410743713378906e-05 nb_pixel_total : 954 time to create 1 rle with old method : 0.0013370513916015625 length of segment : 34 time for calcul the mask position with numpy : 2.8371810913085938e-05 nb_pixel_total : 16 time to create 1 rle with old method : 4.696846008300781e-05 length of segment : 6 time for calcul the mask position with numpy : 2.9802322387695312e-05 nb_pixel_total : 115 time to create 1 rle with old method : 0.00018668174743652344 length of segment : 15 time for calcul the mask position with numpy : 4.6253204345703125e-05 nb_pixel_total : 1111 time to create 1 rle with old method : 0.0015599727630615234 length of segment : 40 Processing 1 images image shape: (400, 400, 3) min: 1.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -100.21172 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 4 time for calcul the mask position with numpy : 6.389617919921875e-05 nb_pixel_total : 1298 time to create 1 rle with old method : 0.0020656585693359375 length of segment : 50 time for calcul the mask position with numpy : 8.416175842285156e-05 nb_pixel_total : 4368 time to create 1 rle with old method : 0.005802631378173828 length of segment : 54 time for calcul the mask position with numpy : 3.814697265625e-05 nb_pixel_total : 470 time to create 1 rle with old method : 0.0007312297821044922 length of segment : 13 time for calcul the mask position with numpy : 3.600120544433594e-05 nb_pixel_total : 475 time to create 1 rle with old method : 0.0007817745208740234 length of segment : 14 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.44609 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 2 time for calcul the mask position with numpy : 4.553794860839844e-05 nb_pixel_total : 223 time to create 1 rle with old method : 0.00039958953857421875 length of segment : 22 time for calcul the mask position with numpy : 4.4345855712890625e-05 nb_pixel_total : 409 time to create 1 rle with old method : 0.0006277561187744141 length of segment : 38 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 239.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 122.30312 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 4 time for calcul the mask position with numpy : 5.14984130859375e-05 nb_pixel_total : 536 time to create 1 rle with old method : 0.0009913444519042969 length of segment : 18 time for calcul the mask position with numpy : 3.981590270996094e-05 nb_pixel_total : 163 time to create 1 rle with old method : 0.0003364086151123047 length of segment : 13 time for calcul the mask position with numpy : 5.459785461425781e-05 nb_pixel_total : 583 time to create 1 rle with old method : 0.0011696815490722656 length of segment : 45 time for calcul the mask position with numpy : 0.0006494522094726562 nb_pixel_total : 65470 time to create 1 rle with old method : 0.07922697067260742 length of segment : 268 Processing 1 images image shape: (400, 400, 3) min: 27.00000 max: 202.00000 molded_images shape: (1, 640, 640, 3) min: -94.29766 max: 84.45156 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 0 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -114.81328 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 9 time for calcul the mask position with numpy : 7.128715515136719e-05 nb_pixel_total : 2383 time to create 1 rle with old method : 0.003247499465942383 length of segment : 50 time for calcul the mask position with numpy : 3.3855438232421875e-05 nb_pixel_total : 103 time to create 1 rle with old method : 0.0002605915069580078 length of segment : 18 time for calcul the mask position with numpy : 4.076957702636719e-05 nb_pixel_total : 701 time to create 1 rle with old method : 0.0010890960693359375 length of segment : 32 time for calcul the mask position with numpy : 3.7670135498046875e-05 nb_pixel_total : 451 time to create 1 rle with old method : 0.0006699562072753906 length of segment : 20 time for calcul the mask position with numpy : 3.838539123535156e-05 nb_pixel_total : 516 time to create 1 rle with old method : 0.0007205009460449219 length of segment : 31 time for calcul the mask position with numpy : 3.147125244140625e-05 nb_pixel_total : 141 time to create 1 rle with old method : 0.00026106834411621094 length of segment : 15 time for calcul the mask position with numpy : 3.0517578125e-05 nb_pixel_total : 107 time to create 1 rle with old method : 0.00022172927856445312 length of segment : 10 time for calcul the mask position with numpy : 3.123283386230469e-05 nb_pixel_total : 139 time to create 1 rle with old method : 0.0002598762512207031 length of segment : 17 time for calcul the mask position with numpy : 3.528594970703125e-05 nb_pixel_total : 453 time to create 1 rle with old method : 0.00067901611328125 length of segment : 19 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -121.87188 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 5 time for calcul the mask position with numpy : 9.226799011230469e-05 nb_pixel_total : 911 time to create 1 rle with old method : 0.0018374919891357422 length of segment : 24 time for calcul the mask position with numpy : 4.863739013671875e-05 nb_pixel_total : 724 time to create 1 rle with old method : 0.001255035400390625 length of segment : 31 time for calcul the mask position with numpy : 3.981590270996094e-05 nb_pixel_total : 128 time to create 1 rle with old method : 0.00029921531677246094 length of segment : 12 time for calcul the mask position with numpy : 3.1948089599609375e-05 nb_pixel_total : 181 time to create 1 rle with old method : 0.0002892017364501953 length of segment : 18 time for calcul the mask position with numpy : 3.886222839355469e-05 nb_pixel_total : 499 time to create 1 rle with old method : 0.0007488727569580078 length of segment : 21 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -120.39922 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 4 time for calcul the mask position with numpy : 4.7206878662109375e-05 nb_pixel_total : 766 time to create 1 rle with old method : 0.0010180473327636719 length of segment : 37 time for calcul the mask position with numpy : 4.839897155761719e-05 nb_pixel_total : 1434 time to create 1 rle with old method : 0.0018949508666992188 length of segment : 43 time for calcul the mask position with numpy : 4.76837158203125e-05 nb_pixel_total : 1306 time to create 1 rle with old method : 0.001749277114868164 length of segment : 40 time for calcul the mask position with numpy : 0.00032711029052734375 nb_pixel_total : 8660 time to create 1 rle with old method : 0.011050224304199219 length of segment : 188 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 254.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 113.02578 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 11 time for calcul the mask position with numpy : 5.936622619628906e-05 nb_pixel_total : 496 time to create 1 rle with old method : 0.0007147789001464844 length of segment : 61 time for calcul the mask position with numpy : 7.224082946777344e-05 nb_pixel_total : 1037 time to create 1 rle with old method : 0.0015141963958740234 length of segment : 102 time for calcul the mask position with numpy : 6.318092346191406e-05 nb_pixel_total : 1245 time to create 1 rle with old method : 0.0017151832580566406 length of segment : 68 time for calcul the mask position with numpy : 3.62396240234375e-05 nb_pixel_total : 298 time to create 1 rle with old method : 0.000423431396484375 length of segment : 29 time for calcul the mask position with numpy : 3.4332275390625e-05 nb_pixel_total : 225 time to create 1 rle with old method : 0.0003418922424316406 length of segment : 31 time for calcul the mask position with numpy : 4.220008850097656e-05 nb_pixel_total : 782 time to create 1 rle with old method : 0.0010793209075927734 length of segment : 42 time for calcul the mask position with numpy : 4.315376281738281e-05 nb_pixel_total : 701 time to create 1 rle with old method : 0.0010170936584472656 length of segment : 32 time for calcul the mask position with numpy : 0.00018596649169921875 nb_pixel_total : 4263 time to create 1 rle with old method : 0.005523204803466797 length of segment : 220 time for calcul the mask position with numpy : 4.506111145019531e-05 nb_pixel_total : 724 time to create 1 rle with old method : 0.0009546279907226562 length of segment : 45 time for calcul the mask position with numpy : 4.482269287109375e-05 nb_pixel_total : 717 time to create 1 rle with old method : 0.0010263919830322266 length of segment : 33 time for calcul the mask position with numpy : 0.0001614093780517578 nb_pixel_total : 3628 time to create 1 rle with old method : 0.004551887512207031 length of segment : 180 Processing 1 images image shape: (280, 400, 3) min: 12.00000 max: 192.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 76.16250 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 1 time for calcul the mask position with numpy : 0.0009362697601318359 nb_pixel_total : 103537 time to create 1 rle with old method : 0.12944674491882324 length of segment : 283 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 150.96328 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 9 time for calcul the mask position with numpy : 4.673004150390625e-05 nb_pixel_total : 860 time to create 1 rle with old method : 0.0011665821075439453 length of segment : 39 time for calcul the mask position with numpy : 4.1484832763671875e-05 nb_pixel_total : 607 time to create 1 rle with old method : 0.0008838176727294922 length of segment : 33 time for calcul the mask position with numpy : 9.441375732421875e-05 nb_pixel_total : 4031 time to create 1 rle with old method : 0.00534367561340332 length of segment : 95 time for calcul the mask position with numpy : 3.814697265625e-05 nb_pixel_total : 417 time to create 1 rle with old method : 0.0006163120269775391 length of segment : 34 time for calcul the mask position with numpy : 6.914138793945312e-05 nb_pixel_total : 3015 time to create 1 rle with old method : 0.0039882659912109375 length of segment : 49 time for calcul the mask position with numpy : 9.131431579589844e-05 nb_pixel_total : 4300 time to create 1 rle with old method : 0.005753278732299805 length of segment : 96 time for calcul the mask position with numpy : 3.552436828613281e-05 nb_pixel_total : 118 time to create 1 rle with old method : 0.0001850128173828125 length of segment : 23 time for calcul the mask position with numpy : 5.1021575927734375e-05 nb_pixel_total : 1692 time to create 1 rle with old method : 0.002246379852294922 length of segment : 39 time for calcul the mask position with numpy : 0.00011348724365234375 nb_pixel_total : 3744 time to create 1 rle with old method : 0.0046460628509521484 length of segment : 62 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 139.63906 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 8 time for calcul the mask position with numpy : 5.221366882324219e-05 nb_pixel_total : 184 time to create 1 rle with old method : 0.00041604042053222656 length of segment : 17 time for calcul the mask position with numpy : 4.2438507080078125e-05 nb_pixel_total : 260 time to create 1 rle with old method : 0.0005202293395996094 length of segment : 20 time for calcul the mask position with numpy : 4.124641418457031e-05 nb_pixel_total : 574 time to create 1 rle with old method : 0.0008451938629150391 length of segment : 34 time for calcul the mask position with numpy : 3.528594970703125e-05 nb_pixel_total : 269 time to create 1 rle with old method : 0.00043272972106933594 length of segment : 17 time for calcul the mask position with numpy : 3.5762786865234375e-05 nb_pixel_total : 289 time to create 1 rle with old method : 0.00041103363037109375 length of segment : 22 time for calcul the mask position with numpy : 4.1484832763671875e-05 nb_pixel_total : 658 time to create 1 rle with old method : 0.0008668899536132812 length of segment : 54 time for calcul the mask position with numpy : 3.719329833984375e-05 nb_pixel_total : 622 time to create 1 rle with old method : 0.0008230209350585938 length of segment : 49 time for calcul the mask position with numpy : 2.9087066650390625e-05 nb_pixel_total : 50 time to create 1 rle with old method : 0.00010824203491210938 length of segment : 6 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 149.70547 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 5 time for calcul the mask position with numpy : 5.4836273193359375e-05 nb_pixel_total : 429 time to create 1 rle with old method : 0.0008695125579833984 length of segment : 22 time for calcul the mask position with numpy : 4.124641418457031e-05 nb_pixel_total : 182 time to create 1 rle with old method : 0.0003902912139892578 length of segment : 22 time for calcul the mask position with numpy : 4.3392181396484375e-05 nb_pixel_total : 254 time to create 1 rle with old method : 0.0005135536193847656 length of segment : 26 time for calcul the mask position with numpy : 7.176399230957031e-05 nb_pixel_total : 27 time to create 1 rle with old method : 0.00026726722717285156 length of segment : 14 time for calcul the mask position with numpy : 4.100799560546875e-05 nb_pixel_total : 213 time to create 1 rle with old method : 0.00041556358337402344 length of segment : 22 Processing 1 images image shape: (280, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 145.47500 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 2 time for calcul the mask position with numpy : 7.414817810058594e-05 nb_pixel_total : 78 time to create 1 rle with old method : 0.0002110004425048828 length of segment : 26 time for calcul the mask position with numpy : 0.00048351287841796875 nb_pixel_total : 34785 time to create 1 rle with old method : 0.04700613021850586 length of segment : 248 NEW PHOTO pour l'instant on ne peut pas sauvegarder la photo dans les tile Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -119.30547 max: 149.88516 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 9 time for calcul the mask position with numpy : 4.839897155761719e-05 nb_pixel_total : 699 time to create 1 rle with old method : 0.001031637191772461 length of segment : 53 time for calcul the mask position with numpy : 3.7670135498046875e-05 nb_pixel_total : 458 time to create 1 rle with old method : 0.0006191730499267578 length of segment : 27 time for calcul the mask position with numpy : 4.076957702636719e-05 nb_pixel_total : 664 time to create 1 rle with old method : 0.000982046127319336 length of segment : 51 time for calcul the mask position with numpy : 3.409385681152344e-05 nb_pixel_total : 257 time to create 1 rle with old method : 0.00037360191345214844 length of segment : 40 time for calcul the mask position with numpy : 4.291534423828125e-05 nb_pixel_total : 804 time to create 1 rle with old method : 0.0011565685272216797 length of segment : 34 time for calcul the mask position with numpy : 3.9577484130859375e-05 nb_pixel_total : 760 time to create 1 rle with old method : 0.0010449886322021484 length of segment : 38 time for calcul the mask position with numpy : 4.1484832763671875e-05 nb_pixel_total : 835 time to create 1 rle with old method : 0.0011827945709228516 length of segment : 39 time for calcul the mask position with numpy : 4.4345855712890625e-05 nb_pixel_total : 1064 time to create 1 rle with old method : 0.0015194416046142578 length of segment : 34 time for calcul the mask position with numpy : 3.170967102050781e-05 nb_pixel_total : 133 time to create 1 rle with old method : 0.00022840499877929688 length of segment : 14 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -118.86406 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 10 time for calcul the mask position with numpy : 7.009506225585938e-05 nb_pixel_total : 2637 time to create 1 rle with old method : 0.003306865692138672 length of segment : 52 time for calcul the mask position with numpy : 3.647804260253906e-05 nb_pixel_total : 304 time to create 1 rle with old method : 0.0004801750183105469 length of segment : 17 time for calcul the mask position with numpy : 0.000171661376953125 nb_pixel_total : 1050 time to create 1 rle with old method : 0.001537322998046875 length of segment : 105 time for calcul the mask position with numpy : 8.368492126464844e-05 nb_pixel_total : 3287 time to create 1 rle with old method : 0.004416704177856445 length of segment : 51 time for calcul the mask position with numpy : 3.4809112548828125e-05 nb_pixel_total : 148 time to create 1 rle with old method : 0.00024199485778808594 length of segment : 11 time for calcul the mask position with numpy : 6.031990051269531e-05 nb_pixel_total : 2061 time to create 1 rle with old method : 0.005973339080810547 length of segment : 85 time for calcul the mask position with numpy : 4.863739013671875e-05 nb_pixel_total : 596 time to create 1 rle with old method : 0.0009009838104248047 length of segment : 33 time for calcul the mask position with numpy : 3.337860107421875e-05 nb_pixel_total : 376 time to create 1 rle with old method : 0.0005731582641601562 length of segment : 18 time for calcul the mask position with numpy : 4.267692565917969e-05 nb_pixel_total : 467 time to create 1 rle with old method : 0.0007317066192626953 length of segment : 29 time for calcul the mask position with numpy : 9.369850158691406e-05 nb_pixel_total : 3585 time to create 1 rle with old method : 0.004933834075927734 length of segment : 47 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -103.96172 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 3 time for calcul the mask position with numpy : 8.416175842285156e-05 nb_pixel_total : 3852 time to create 1 rle with old method : 0.005158424377441406 length of segment : 50 time for calcul the mask position with numpy : 6.008148193359375e-05 nb_pixel_total : 493 time to create 1 rle with old method : 0.0008275508880615234 length of segment : 15 time for calcul the mask position with numpy : 3.6716461181640625e-05 nb_pixel_total : 460 time to create 1 rle with old method : 0.0007407665252685547 length of segment : 12 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.34453 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 2 time for calcul the mask position with numpy : 3.814697265625e-05 nb_pixel_total : 216 time to create 1 rle with old method : 0.0003173351287841797 length of segment : 22 time for calcul the mask position with numpy : 3.647804260253906e-05 nb_pixel_total : 442 time to create 1 rle with old method : 0.0005676746368408203 length of segment : 43 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 237.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 118.06484 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 3 time for calcul the mask position with numpy : 4.1484832763671875e-05 nb_pixel_total : 494 time to create 1 rle with old method : 0.0007572174072265625 length of segment : 17 time for calcul the mask position with numpy : 0.0006573200225830078 nb_pixel_total : 64671 time to create 1 rle with old method : 0.08080887794494629 length of segment : 292 time for calcul the mask position with numpy : 5.698204040527344e-05 nb_pixel_total : 508 time to create 1 rle with old method : 0.0006740093231201172 length of segment : 56 Processing 1 images image shape: (400, 400, 3) min: 21.00000 max: 198.00000 molded_images shape: (1, 640, 640, 3) min: -92.21953 max: 86.73672 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 0 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -114.55938 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 6 time for calcul the mask position with numpy : 4.315376281738281e-05 nb_pixel_total : 152 time to create 1 rle with old method : 0.0002460479736328125 length of segment : 15 time for calcul the mask position with numpy : 6.4849853515625e-05 nb_pixel_total : 2408 time to create 1 rle with old method : 0.003067493438720703 length of segment : 51 time for calcul the mask position with numpy : 3.2901763916015625e-05 nb_pixel_total : 134 time to create 1 rle with old method : 0.0002086162567138672 length of segment : 17 time for calcul the mask position with numpy : 3.528594970703125e-05 nb_pixel_total : 421 time to create 1 rle with old method : 0.000637054443359375 length of segment : 20 time for calcul the mask position with numpy : 5.6743621826171875e-05 nb_pixel_total : 1338 time to create 1 rle with old method : 0.0018358230590820312 length of segment : 61 time for calcul the mask position with numpy : 3.62396240234375e-05 nb_pixel_total : 393 time to create 1 rle with old method : 0.0005440711975097656 length of segment : 25 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -111.00469 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 7 time for calcul the mask position with numpy : 4.4345855712890625e-05 nb_pixel_total : 711 time to create 1 rle with old method : 0.0009005069732666016 length of segment : 31 time for calcul the mask position with numpy : 3.981590270996094e-05 nb_pixel_total : 310 time to create 1 rle with old method : 0.0004482269287109375 length of segment : 38 time for calcul the mask position with numpy : 6.198883056640625e-05 nb_pixel_total : 1421 time to create 1 rle with old method : 0.0021066665649414062 length of segment : 41 time for calcul the mask position with numpy : 3.409385681152344e-05 nb_pixel_total : 152 time to create 1 rle with old method : 0.0002758502960205078 length of segment : 13 time for calcul the mask position with numpy : 3.814697265625e-05 nb_pixel_total : 733 time to create 1 rle with old method : 0.0009996891021728516 length of segment : 25 time for calcul the mask position with numpy : 3.933906555175781e-05 nb_pixel_total : 463 time to create 1 rle with old method : 0.0007483959197998047 length of segment : 19 time for calcul the mask position with numpy : 4.2438507080078125e-05 nb_pixel_total : 765 time to create 1 rle with old method : 0.0011687278747558594 length of segment : 34 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -118.01250 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 6 time for calcul the mask position with numpy : 4.5299530029296875e-05 nb_pixel_total : 720 time to create 1 rle with old method : 0.00092315673828125 length of segment : 36 time for calcul the mask position with numpy : 4.458427429199219e-05 nb_pixel_total : 1307 time to create 1 rle with old method : 0.0016751289367675781 length of segment : 40 time for calcul the mask position with numpy : 4.506111145019531e-05 nb_pixel_total : 948 time to create 1 rle with old method : 0.001386404037475586 length of segment : 31 time for calcul the mask position with numpy : 4.2438507080078125e-05 nb_pixel_total : 1414 time to create 1 rle with old method : 0.0017852783203125 length of segment : 46 time for calcul the mask position with numpy : 0.00018286705017089844 nb_pixel_total : 5211 time to create 1 rle with old method : 0.006983757019042969 length of segment : 184 time for calcul the mask position with numpy : 0.0003523826599121094 nb_pixel_total : 9984 time to create 1 rle with old method : 0.01225590705871582 length of segment : 206 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 251.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 110.40078 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 11 time for calcul the mask position with numpy : 5.7697296142578125e-05 nb_pixel_total : 508 time to create 1 rle with old method : 0.0007412433624267578 length of segment : 68 time for calcul the mask position with numpy : 3.4332275390625e-05 nb_pixel_total : 220 time to create 1 rle with old method : 0.00033855438232421875 length of segment : 31 time for calcul the mask position with numpy : 6.0558319091796875e-05 nb_pixel_total : 1461 time to create 1 rle with old method : 0.002061605453491211 length of segment : 95 time for calcul the mask position with numpy : 4.291534423828125e-05 nb_pixel_total : 746 time to create 1 rle with old method : 0.0009591579437255859 length of segment : 44 time for calcul the mask position with numpy : 5.7697296142578125e-05 nb_pixel_total : 595 time to create 1 rle with old method : 0.0008671283721923828 length of segment : 76 time for calcul the mask position with numpy : 3.4332275390625e-05 nb_pixel_total : 291 time to create 1 rle with old method : 0.0004029273986816406 length of segment : 30 time for calcul the mask position with numpy : 5.817413330078125e-05 nb_pixel_total : 1497 time to create 1 rle with old method : 0.002110004425048828 length of segment : 96 time for calcul the mask position with numpy : 4.9114227294921875e-05 nb_pixel_total : 310 time to create 1 rle with old method : 0.0005743503570556641 length of segment : 31 time for calcul the mask position with numpy : 0.00021767616271972656 nb_pixel_total : 3874 time to create 1 rle with old method : 0.0073032379150390625 length of segment : 193 time for calcul the mask position with numpy : 4.76837158203125e-05 nb_pixel_total : 652 time to create 1 rle with old method : 0.0009918212890625 length of segment : 30 time for calcul the mask position with numpy : 4.4345855712890625e-05 nb_pixel_total : 757 time to create 1 rle with old method : 0.001004934310913086 length of segment : 45 Processing 1 images image shape: (280, 400, 3) min: 20.00000 max: 193.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 79.88516 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 0 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 9 time for calcul the mask position with numpy : 0.00011229515075683594 nb_pixel_total : 3702 time to create 1 rle with old method : 0.0064542293548583984 length of segment : 94 time for calcul the mask position with numpy : 4.00543212890625e-05 nb_pixel_total : 198 time to create 1 rle with old method : 0.00037384033203125 length of segment : 29 time for calcul the mask position with numpy : 4.2438507080078125e-05 nb_pixel_total : 340 time to create 1 rle with old method : 0.0006718635559082031 length of segment : 26 time for calcul the mask position with numpy : 8.368492126464844e-05 nb_pixel_total : 2901 time to create 1 rle with old method : 0.005189418792724609 length of segment : 45 time for calcul the mask position with numpy : 7.200241088867188e-05 nb_pixel_total : 539 time to create 1 rle with old method : 0.001332998275756836 length of segment : 31 time for calcul the mask position with numpy : 7.987022399902344e-05 nb_pixel_total : 782 time to create 1 rle with old method : 0.0019042491912841797 length of segment : 43 time for calcul the mask position with numpy : 5.245208740234375e-05 nb_pixel_total : 1658 time to create 1 rle with old method : 0.002241849899291992 length of segment : 41 time for calcul the mask position with numpy : 9.584426879882812e-05 nb_pixel_total : 4053 time to create 1 rle with old method : 0.004960775375366211 length of segment : 92 time for calcul the mask position with numpy : 4.291534423828125e-05 nb_pixel_total : 849 time to create 1 rle with old method : 0.0010573863983154297 length of segment : 39 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 137.81484 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 9 time for calcul the mask position with numpy : 3.886222839355469e-05 nb_pixel_total : 264 time to create 1 rle with old method : 0.0003864765167236328 length of segment : 19 time for calcul the mask position with numpy : 3.24249267578125e-05 nb_pixel_total : 183 time to create 1 rle with old method : 0.0002906322479248047 length of segment : 17 time for calcul the mask position with numpy : 3.600120544433594e-05 nb_pixel_total : 696 time to create 1 rle with old method : 0.0009546279907226562 length of segment : 35 time for calcul the mask position with numpy : 2.9087066650390625e-05 nb_pixel_total : 244 time to create 1 rle with old method : 0.00033855438232421875 length of segment : 15 time for calcul the mask position with numpy : 3.218650817871094e-05 nb_pixel_total : 269 time to create 1 rle with old method : 0.0003933906555175781 length of segment : 21 time for calcul the mask position with numpy : 3.361701965332031e-05 nb_pixel_total : 560 time to create 1 rle with old method : 0.0006890296936035156 length of segment : 42 time for calcul the mask position with numpy : 3.266334533691406e-05 nb_pixel_total : 274 time to create 1 rle with old method : 0.00044035911560058594 length of segment : 16 time for calcul the mask position with numpy : 3.266334533691406e-05 nb_pixel_total : 278 time to create 1 rle with old method : 0.0004303455352783203 length of segment : 17 time for calcul the mask position with numpy : 2.86102294921875e-05 nb_pixel_total : 260 time to create 1 rle with old method : 0.00035309791564941406 length of segment : 21 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 149.43984 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 3 time for calcul the mask position with numpy : 4.792213439941406e-05 nb_pixel_total : 311 time to create 1 rle with old method : 0.0005419254302978516 length of segment : 40 time for calcul the mask position with numpy : 3.3855438232421875e-05 nb_pixel_total : 469 time to create 1 rle with old method : 0.0006935596466064453 length of segment : 24 time for calcul the mask position with numpy : 3.123283386230469e-05 nb_pixel_total : 185 time to create 1 rle with old method : 0.0002837181091308594 length of segment : 22 Processing 1 images image shape: (280, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 142.91250 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 3 time for calcul the mask position with numpy : 4.2438507080078125e-05 nb_pixel_total : 49 time to create 1 rle with old method : 0.00011968612670898438 length of segment : 8 time for calcul the mask position with numpy : 2.8371810913085938e-05 nb_pixel_total : 21 time to create 1 rle with old method : 6.079673767089844e-05 length of segment : 5 time for calcul the mask position with numpy : 0.00042819976806640625 nb_pixel_total : 36096 time to create 1 rle with old method : 0.04351186752319336 length of segment : 213 Detection mask done ! Trying to reset tf kernel 3813860 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5525 tf kernel not reseted sub process len(results) : 166 len(list_Values) 166 None max_time_sub_proc : 3600 parent process len(results) : 0 len(list_Values) 166 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 10814 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Catched exception ! Connect or reconnect ! Number saved : None batch 1 Loaded 166 chid ids of type : 4228 Number RLEs to save : 7968 save missing photos in datou_result : time spend for datou_step_exec : 21.224348783493042 time spend to save output : 1.5258424282073975 total time spend for step 1 : 22.75019121170044 step2:brightness Mon Mar 3 01:20:54 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure inside step calcul brightness treat image : temp/1740961231_3813386_1340531384_9e60ef5549cedc3caf4a1506efced53d.jpg treat image : temp/1740961231_3813386_1340531370_3297724219f4830fa3cb41e5c7d06705.jpg Inside saveOutput : final : False verbose : 0 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 2 time used for this insertion : 0.012929439544677734 begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 2 time used for this insertion : 0.01074075698852539 save missing photos in datou_result : time spend for datou_step_exec : 0.9126505851745605 time spend to save output : 0.027879953384399414 total time spend for step 2 : 0.94053053855896 step3:blur_detection Mon Mar 3 01:20:55 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure inside step blur_detection methode: ratio et variance treat image : temp/1740961231_3813386_1340531384_9e60ef5549cedc3caf4a1506efced53d.jpg resize: (1080, 1920) 1340531384 -7.501900442709465 treat image : temp/1740961231_3813386_1340531370_3297724219f4830fa3cb41e5c7d06705.jpg resize: (1080, 1920) 1340531370 -7.500526182773834 Inside saveOutput : final : False verbose : 0 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 2 time used for this insertion : 0.012929201126098633 begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 2 time used for this insertion : 0.011104822158813477 save missing photos in datou_result : time spend for datou_step_exec : 1.9297983646392822 time spend to save output : 0.029581308364868164 total time spend for step 3 : 1.9593796730041504 step4:rle_unique_nms_with_priority Mon Mar 3 01:20:57 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms batch 1 Loaded 166 chid ids of type : 4228 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++nb_obj : 85 nb_hashtags : 6 time to prepare the origin masks : 1.0330615043640137 time for calcul the mask position with numpy : 0.06085324287414551 nb_pixel_total : 1804749 time to create 1 rle with new method : 0.2120661735534668 time for calcul the mask position with numpy : 0.008717060089111328 nb_pixel_total : 65470 time to create 1 rle with old method : 0.07404088973999023 time for calcul the mask position with numpy : 0.008159637451171875 nb_pixel_total : 70 time to create 1 rle with old method : 0.00010037422180175781 time for calcul the mask position with numpy : 0.008020162582397461 nb_pixel_total : 464 time to create 1 rle with old method : 0.0005664825439453125 time for calcul the mask position with numpy : 0.008344173431396484 nb_pixel_total : 49 time to create 1 rle with old method : 9.632110595703125e-05 time for calcul the mask position with numpy : 0.008384466171264648 nb_pixel_total : 45 time to create 1 rle with old method : 8.082389831542969e-05 time for calcul the mask position with numpy : 0.008237600326538086 nb_pixel_total : 166 time to create 1 rle with old method : 0.0002276897430419922 time for calcul the mask position with numpy : 0.008224010467529297 nb_pixel_total : 462 time to create 1 rle with old method : 0.0005345344543457031 time for calcul the mask position with numpy : 0.00812983512878418 nb_pixel_total : 1156 time to create 1 rle with old method : 0.0013720989227294922 time for calcul the mask position with numpy : 0.008324623107910156 nb_pixel_total : 623 time to create 1 rle with old method : 0.0007536411285400391 time for calcul the mask position with numpy : 0.008217334747314453 nb_pixel_total : 103 time to create 1 rle with old method : 0.0001780986785888672 time for calcul the mask position with numpy : 0.008183956146240234 nb_pixel_total : 2506 time to create 1 rle with old method : 0.002750873565673828 time for calcul the mask position with numpy : 0.008126497268676758 nb_pixel_total : 1298 time to create 1 rle with old method : 0.0014982223510742188 time for calcul the mask position with numpy : 0.007995843887329102 nb_pixel_total : 875 time to create 1 rle with old method : 0.0010485649108886719 time for calcul the mask position with numpy : 0.00827646255493164 nb_pixel_total : 193 time to create 1 rle with old method : 0.0002124309539794922 time for calcul the mask position with numpy : 0.008144378662109375 nb_pixel_total : 166 time to create 1 rle with old method : 0.00019979476928710938 time for calcul the mask position with numpy : 0.008275032043457031 nb_pixel_total : 2 time to create 1 rle with old method : 1.1682510375976562e-05 time for calcul the mask position with numpy : 0.007949113845825195 nb_pixel_total : 223 time to create 1 rle with old method : 0.0002570152282714844 time for calcul the mask position with numpy : 0.00825047492980957 nb_pixel_total : 29 time to create 1 rle with old method : 3.886222839355469e-05 time for calcul the mask position with numpy : 0.008094310760498047 nb_pixel_total : 3509 time to create 1 rle with old method : 0.004057884216308594 time for calcul the mask position with numpy : 0.00806570053100586 nb_pixel_total : 115 time to create 1 rle with old method : 0.00015020370483398438 time for calcul the mask position with numpy : 0.007907867431640625 nb_pixel_total : 866 time to create 1 rle with old method : 0.0012083053588867188 time for calcul the mask position with numpy : 0.009632110595703125 nb_pixel_total : 4368 time to create 1 rle with old method : 0.007303714752197266 time for calcul the mask position with numpy : 0.009590864181518555 nb_pixel_total : 409 time to create 1 rle with old method : 0.0007119178771972656 time for calcul the mask position with numpy : 0.00958561897277832 nb_pixel_total : 23 time to create 1 rle with old method : 0.00010585784912109375 time for calcul the mask position with numpy : 0.009583711624145508 nb_pixel_total : 470 time to create 1 rle with old method : 0.0008323192596435547 time for calcul the mask position with numpy : 0.009602785110473633 nb_pixel_total : 163 time to create 1 rle with old method : 0.00029397010803222656 time for calcul the mask position with numpy : 0.009622812271118164 nb_pixel_total : 98 time to create 1 rle with old method : 0.00017690658569335938 time for calcul the mask position with numpy : 0.00953221321105957 nb_pixel_total : 583 time to create 1 rle with old method : 0.0007116794586181641 time for calcul the mask position with numpy : 0.008391618728637695 nb_pixel_total : 169 time to create 1 rle with old method : 0.0004169940948486328 time for calcul the mask position with numpy : 0.008090496063232422 nb_pixel_total : 954 time to create 1 rle with old method : 0.0011403560638427734 time for calcul the mask position with numpy : 0.00816798210144043 nb_pixel_total : 536 time to create 1 rle with old method : 0.0006728172302246094 time for calcul the mask position with numpy : 0.008131742477416992 nb_pixel_total : 107 time to create 1 rle with old method : 0.00013303756713867188 time for calcul the mask position with numpy : 0.008211612701416016 nb_pixel_total : 141 time to create 1 rle with old method : 0.0001888275146484375 time for calcul the mask position with numpy : 0.008261919021606445 nb_pixel_total : 7 time to create 1 rle with old method : 2.765655517578125e-05 time for calcul the mask position with numpy : 0.008385181427001953 nb_pixel_total : 724 time to create 1 rle with old method : 0.0008482933044433594 time for calcul the mask position with numpy : 0.007961273193359375 nb_pixel_total : 766 time to create 1 rle with old method : 0.0009062290191650391 time for calcul the mask position with numpy : 0.008174896240234375 nb_pixel_total : 128 time to create 1 rle with old method : 0.00016260147094726562 time for calcul the mask position with numpy : 0.00827646255493164 nb_pixel_total : 225 time to create 1 rle with old method : 0.00027871131896972656 time for calcul the mask position with numpy : 0.008319377899169922 nb_pixel_total : 1434 time to create 1 rle with old method : 0.001621246337890625 time for calcul the mask position with numpy : 0.008306264877319336 nb_pixel_total : 782 time to create 1 rle with old method : 0.0009303092956542969 time for calcul the mask position with numpy : 0.008424043655395508 nb_pixel_total : 1 time to create 1 rle with old method : 8.58306884765625e-06 time for calcul the mask position with numpy : 0.008307218551635742 nb_pixel_total : 298 time to create 1 rle with old method : 0.00035381317138671875 time for calcul the mask position with numpy : 0.008170843124389648 nb_pixel_total : 1 time to create 1 rle with old method : 7.867813110351562e-06 time for calcul the mask position with numpy : 0.008162260055541992 nb_pixel_total : 41 time to create 1 rle with old method : 7.605552673339844e-05 time for calcul the mask position with numpy : 0.008153676986694336 nb_pixel_total : 701 time to create 1 rle with old method : 0.0008275508880615234 time for calcul the mask position with numpy : 0.008262395858764648 nb_pixel_total : 911 time to create 1 rle with old method : 0.0010056495666503906 time for calcul the mask position with numpy : 0.007970333099365234 nb_pixel_total : 4263 time to create 1 rle with old method : 0.004797220230102539 time for calcul the mask position with numpy : 0.00818181037902832 nb_pixel_total : 8660 time to create 1 rle with old method : 0.009902715682983398 time for calcul the mask position with numpy : 0.008216619491577148 nb_pixel_total : 496 time to create 1 rle with old method : 0.0005633831024169922 time for calcul the mask position with numpy : 0.00817728042602539 nb_pixel_total : 462 time to create 1 rle with old method : 0.0006084442138671875 time for calcul the mask position with numpy : 0.008120536804199219 nb_pixel_total : 1037 time to create 1 rle with old method : 0.0012276172637939453 time for calcul the mask position with numpy : 0.008205413818359375 nb_pixel_total : 499 time to create 1 rle with old method : 0.0005991458892822266 time for calcul the mask position with numpy : 0.009769678115844727 nb_pixel_total : 258 time to create 1 rle with old method : 0.0003948211669921875 time for calcul the mask position with numpy : 0.00841212272644043 nb_pixel_total : 516 time to create 1 rle with old method : 0.0006155967712402344 time for calcul the mask position with numpy : 0.008450746536254883 nb_pixel_total : 701 time to create 1 rle with old method : 0.0009059906005859375 time for calcul the mask position with numpy : 0.008386373519897461 nb_pixel_total : 181 time to create 1 rle with old method : 0.00021958351135253906 time for calcul the mask position with numpy : 0.008443355560302734 nb_pixel_total : 2383 time to create 1 rle with old method : 0.002782583236694336 time for calcul the mask position with numpy : 0.008245706558227539 nb_pixel_total : 451 time to create 1 rle with old method : 0.0005691051483154297 time for calcul the mask position with numpy : 0.008498907089233398 nb_pixel_total : 17 time to create 1 rle with old method : 5.936622619628906e-05 time for calcul the mask position with numpy : 0.009811878204345703 nb_pixel_total : 103 time to create 1 rle with old method : 0.00013637542724609375 time for calcul the mask position with numpy : 0.009263992309570312 nb_pixel_total : 103537 time to create 1 rle with old method : 0.12050652503967285 time for calcul the mask position with numpy : 0.008436918258666992 nb_pixel_total : 78 time to create 1 rle with old method : 0.00010776519775390625 time for calcul the mask position with numpy : 0.01203775405883789 nb_pixel_total : 429 time to create 1 rle with old method : 0.0007612705230712891 time for calcul the mask position with numpy : 0.009375572204589844 nb_pixel_total : 118 time to create 1 rle with old method : 0.00016260147094726562 time for calcul the mask position with numpy : 0.008673667907714844 nb_pixel_total : 34785 time to create 1 rle with old method : 0.04412722587585449 time for calcul the mask position with numpy : 0.009375333786010742 nb_pixel_total : 3015 time to create 1 rle with old method : 0.003706693649291992 time for calcul the mask position with numpy : 0.009948492050170898 nb_pixel_total : 658 time to create 1 rle with old method : 0.0008199214935302734 time for calcul the mask position with numpy : 0.008753776550292969 nb_pixel_total : 38 time to create 1 rle with old method : 7.82012939453125e-05 time for calcul the mask position with numpy : 0.008518457412719727 nb_pixel_total : 184 time to create 1 rle with old method : 0.00024318695068359375 time for calcul the mask position with numpy : 0.008539676666259766 nb_pixel_total : 260 time to create 1 rle with old method : 0.0003228187561035156 time for calcul the mask position with numpy : 0.008498430252075195 nb_pixel_total : 4031 time to create 1 rle with old method : 0.004748821258544922 time for calcul the mask position with numpy : 0.008475065231323242 nb_pixel_total : 280 time to create 1 rle with old method : 0.00042319297790527344 time for calcul the mask position with numpy : 0.008505105972290039 nb_pixel_total : 607 time to create 1 rle with old method : 0.0007212162017822266 time for calcul the mask position with numpy : 0.008283138275146484 nb_pixel_total : 1692 time to create 1 rle with old method : 0.0019648075103759766 time for calcul the mask position with numpy : 0.008374214172363281 nb_pixel_total : 860 time to create 1 rle with old method : 0.0009517669677734375 time for calcul the mask position with numpy : 0.008121728897094727 nb_pixel_total : 254 time to create 1 rle with old method : 0.00031638145446777344 time for calcul the mask position with numpy : 0.008375406265258789 nb_pixel_total : 269 time to create 1 rle with old method : 0.0003299713134765625 time for calcul the mask position with numpy : 0.008333206176757812 nb_pixel_total : 289 time to create 1 rle with old method : 0.0003597736358642578 time for calcul the mask position with numpy : 0.008340120315551758 nb_pixel_total : 16 time to create 1 rle with old method : 3.361701965332031e-05 time for calcul the mask position with numpy : 0.008344650268554688 nb_pixel_total : 3744 time to create 1 rle with old method : 0.004535198211669922 time for calcul the mask position with numpy : 0.008314132690429688 nb_pixel_total : 27 time to create 1 rle with old method : 7.486343383789062e-05 time for calcul the mask position with numpy : 0.00821375846862793 nb_pixel_total : 417 time to create 1 rle with old method : 0.0005133152008056641 time for calcul the mask position with numpy : 0.008336782455444336 nb_pixel_total : 574 time to create 1 rle with old method : 0.0006861686706542969 time for calcul the mask position with numpy : 0.008075237274169922 nb_pixel_total : 182 time to create 1 rle with old method : 0.000213623046875 time for calcul the mask position with numpy : 0.008064508438110352 nb_pixel_total : 50 time to create 1 rle with old method : 7.700920104980469e-05 create new chi : 1.328721523284912 after preparing all the mask , begin to delete the rle from the crop_hashtag_id => VR 28-11-20 : il faut déplacer cela apres le save_crop_hashtag_ids_obj de la ligne 9514 ! we have 0 chi objets contains the rles time to delete rle : 0.014265298843383789 batch 1 Loaded 86 chid ids of type : 4230 Number RLEs to save : 8204 TO DO : save crop sub photo not yet done ! save time : 1.6951625347137451 nb_obj : 81 nb_hashtags : 5 time to prepare the origin masks : 1.7853803634643555 time for calcul the mask position with numpy : 0.03276181221008301 nb_pixel_total : 1903642 time to create 1 rle with new method : 0.4408755302429199 time for calcul the mask position with numpy : 0.007076263427734375 nb_pixel_total : 64671 time to create 1 rle with old method : 0.07216215133666992 time for calcul the mask position with numpy : 0.006360054016113281 nb_pixel_total : 82 time to create 1 rle with old method : 0.00012683868408203125 time for calcul the mask position with numpy : 0.00601649284362793 nb_pixel_total : 96 time to create 1 rle with old method : 0.00013971328735351562 time for calcul the mask position with numpy : 0.0060541629791259766 nb_pixel_total : 458 time to create 1 rle with old method : 0.0005097389221191406 time for calcul the mask position with numpy : 0.005902528762817383 nb_pixel_total : 383 time to create 1 rle with old method : 0.0004792213439941406 time for calcul the mask position with numpy : 0.005922555923461914 nb_pixel_total : 15 time to create 1 rle with old method : 4.5299530029296875e-05 time for calcul the mask position with numpy : 0.006037473678588867 nb_pixel_total : 146 time to create 1 rle with old method : 0.0002181529998779297 time for calcul the mask position with numpy : 0.0060024261474609375 nb_pixel_total : 18 time to create 1 rle with old method : 6.318092346191406e-05 time for calcul the mask position with numpy : 0.009357452392578125 nb_pixel_total : 699 time to create 1 rle with old method : 0.0008597373962402344 time for calcul the mask position with numpy : 0.0073621273040771484 nb_pixel_total : 1064 time to create 1 rle with old method : 0.0012924671173095703 time for calcul the mask position with numpy : 0.00896596908569336 nb_pixel_total : 2637 time to create 1 rle with old method : 0.0030062198638916016 time for calcul the mask position with numpy : 0.009227991104125977 nb_pixel_total : 1050 time to create 1 rle with old method : 0.0012869834899902344 time for calcul the mask position with numpy : 0.009380578994750977 nb_pixel_total : 148 time to create 1 rle with old method : 0.00020122528076171875 time for calcul the mask position with numpy : 0.008183717727661133 nb_pixel_total : 304 time to create 1 rle with old method : 0.00038695335388183594 time for calcul the mask position with numpy : 0.00803685188293457 nb_pixel_total : 216 time to create 1 rle with old method : 0.00027632713317871094 time for calcul the mask position with numpy : 0.008925199508666992 nb_pixel_total : 596 time to create 1 rle with old method : 0.0007455348968505859 time for calcul the mask position with numpy : 0.009468317031860352 nb_pixel_total : 549 time to create 1 rle with old method : 0.0006844997406005859 time for calcul the mask position with numpy : 0.00811624526977539 nb_pixel_total : 485 time to create 1 rle with old method : 0.0005388259887695312 time for calcul the mask position with numpy : 0.009356975555419922 nb_pixel_total : 3287 time to create 1 rle with old method : 0.003871917724609375 time for calcul the mask position with numpy : 0.00936579704284668 nb_pixel_total : 2061 time to create 1 rle with old method : 0.0023005008697509766 time for calcul the mask position with numpy : 0.007651805877685547 nb_pixel_total : 1 time to create 1 rle with old method : 2.2172927856445312e-05 time for calcul the mask position with numpy : 0.006092071533203125 nb_pixel_total : 1 time to create 1 rle with old method : 2.0265579223632812e-05 time for calcul the mask position with numpy : 0.0061435699462890625 nb_pixel_total : 3852 time to create 1 rle with old method : 0.00458836555480957 time for calcul the mask position with numpy : 0.0058765411376953125 nb_pixel_total : 442 time to create 1 rle with old method : 0.0005211830139160156 time for calcul the mask position with numpy : 0.005781412124633789 nb_pixel_total : 493 time to create 1 rle with old method : 0.0006105899810791016 time for calcul the mask position with numpy : 0.005857944488525391 nb_pixel_total : 6 time to create 1 rle with old method : 6.67572021484375e-05 time for calcul the mask position with numpy : 0.005770683288574219 nb_pixel_total : 494 time to create 1 rle with old method : 0.0005631446838378906 time for calcul the mask position with numpy : 0.005772590637207031 nb_pixel_total : 5211 time to create 1 rle with old method : 0.006139993667602539 time for calcul the mask position with numpy : 0.005624532699584961 nb_pixel_total : 152 time to create 1 rle with old method : 0.00020074844360351562 time for calcul the mask position with numpy : 0.005872249603271484 nb_pixel_total : 711 time to create 1 rle with old method : 0.0007791519165039062 time for calcul the mask position with numpy : 0.005944967269897461 nb_pixel_total : 1338 time to create 1 rle with old method : 0.0015475749969482422 time for calcul the mask position with numpy : 0.005928993225097656 nb_pixel_total : 720 time to create 1 rle with old method : 0.0008027553558349609 time for calcul the mask position with numpy : 0.0059452056884765625 nb_pixel_total : 152 time to create 1 rle with old method : 0.00019788742065429688 time for calcul the mask position with numpy : 0.005751609802246094 nb_pixel_total : 220 time to create 1 rle with old method : 0.0002803802490234375 time for calcul the mask position with numpy : 0.005703926086425781 nb_pixel_total : 11 time to create 1 rle with old method : 4.172325134277344e-05 time for calcul the mask position with numpy : 0.0059697628021240234 nb_pixel_total : 1421 time to create 1 rle with old method : 0.0016734600067138672 time for calcul the mask position with numpy : 0.005831718444824219 nb_pixel_total : 151 time to create 1 rle with old method : 0.0002181529998779297 time for calcul the mask position with numpy : 0.00571751594543457 nb_pixel_total : 1307 time to create 1 rle with old method : 0.0015647411346435547 time for calcul the mask position with numpy : 0.006005287170410156 nb_pixel_total : 31 time to create 1 rle with old method : 6.794929504394531e-05 time for calcul the mask position with numpy : 0.005833625793457031 nb_pixel_total : 746 time to create 1 rle with old method : 0.0009157657623291016 time for calcul the mask position with numpy : 0.006044626235961914 nb_pixel_total : 291 time to create 1 rle with old method : 0.0003361701965332031 time for calcul the mask position with numpy : 0.006165742874145508 nb_pixel_total : 24 time to create 1 rle with old method : 5.364418029785156e-05 time for calcul the mask position with numpy : 0.005858898162841797 nb_pixel_total : 652 time to create 1 rle with old method : 0.0007846355438232422 time for calcul the mask position with numpy : 0.0060770511627197266 nb_pixel_total : 733 time to create 1 rle with old method : 0.0007948875427246094 time for calcul the mask position with numpy : 0.005770444869995117 nb_pixel_total : 3874 time to create 1 rle with old method : 0.004450082778930664 time for calcul the mask position with numpy : 0.005605459213256836 nb_pixel_total : 9984 time to create 1 rle with old method : 0.010694503784179688 time for calcul the mask position with numpy : 0.005536794662475586 nb_pixel_total : 508 time to create 1 rle with old method : 0.0005741119384765625 time for calcul the mask position with numpy : 0.005510091781616211 nb_pixel_total : 325 time to create 1 rle with old method : 0.0003974437713623047 time for calcul the mask position with numpy : 0.0054857730865478516 nb_pixel_total : 1461 time to create 1 rle with old method : 0.0016579627990722656 time for calcul the mask position with numpy : 0.0055141448974609375 nb_pixel_total : 463 time to create 1 rle with old method : 0.0005497932434082031 time for calcul the mask position with numpy : 0.005568027496337891 nb_pixel_total : 393 time to create 1 rle with old method : 0.0004482269287109375 time for calcul the mask position with numpy : 0.005518198013305664 nb_pixel_total : 310 time to create 1 rle with old method : 0.00035762786865234375 time for calcul the mask position with numpy : 0.005754947662353516 nb_pixel_total : 383 time to create 1 rle with old method : 0.0004916191101074219 time for calcul the mask position with numpy : 0.005605220794677734 nb_pixel_total : 948 time to create 1 rle with old method : 0.0011212825775146484 time for calcul the mask position with numpy : 0.0056934356689453125 nb_pixel_total : 2408 time to create 1 rle with old method : 0.002772092819213867 time for calcul the mask position with numpy : 0.005959749221801758 nb_pixel_total : 421 time to create 1 rle with old method : 0.00048232078552246094 time for calcul the mask position with numpy : 0.005941867828369141 nb_pixel_total : 134 time to create 1 rle with old method : 0.00018405914306640625 time for calcul the mask position with numpy : 0.006101131439208984 nb_pixel_total : 469 time to create 1 rle with old method : 0.0006229877471923828 time for calcul the mask position with numpy : 0.005887269973754883 nb_pixel_total : 198 time to create 1 rle with old method : 0.00023484230041503906 time for calcul the mask position with numpy : 0.0058746337890625 nb_pixel_total : 36047 time to create 1 rle with old method : 0.04053616523742676 time for calcul the mask position with numpy : 0.005875349044799805 nb_pixel_total : 2901 time to create 1 rle with old method : 0.003350973129272461 time for calcul the mask position with numpy : 0.005857944488525391 nb_pixel_total : 560 time to create 1 rle with old method : 0.0006883144378662109 time for calcul the mask position with numpy : 0.005640983581542969 nb_pixel_total : 183 time to create 1 rle with old method : 0.00022125244140625 time for calcul the mask position with numpy : 0.005684614181518555 nb_pixel_total : 264 time to create 1 rle with old method : 0.00039196014404296875 time for calcul the mask position with numpy : 0.005675315856933594 nb_pixel_total : 49 time to create 1 rle with old method : 8.130073547363281e-05 time for calcul the mask position with numpy : 0.005860567092895508 nb_pixel_total : 413 time to create 1 rle with old method : 0.0005755424499511719 time for calcul the mask position with numpy : 0.005579471588134766 nb_pixel_total : 3702 time to create 1 rle with old method : 0.004483222961425781 time for calcul the mask position with numpy : 0.005986213684082031 nb_pixel_total : 527 time to create 1 rle with old method : 0.0006625652313232422 time for calcul the mask position with numpy : 0.005927562713623047 nb_pixel_total : 274 time to create 1 rle with old method : 0.00035381317138671875 time for calcul the mask position with numpy : 0.006135225296020508 nb_pixel_total : 1658 time to create 1 rle with old method : 0.0020093917846679688 time for calcul the mask position with numpy : 0.006198883056640625 nb_pixel_total : 782 time to create 1 rle with old method : 0.0009486675262451172 time for calcul the mask position with numpy : 0.006103038787841797 nb_pixel_total : 311 time to create 1 rle with old method : 0.0003876686096191406 time for calcul the mask position with numpy : 0.0060694217681884766 nb_pixel_total : 90 time to create 1 rle with old method : 0.00014352798461914062 time for calcul the mask position with numpy : 0.0073773860931396484 nb_pixel_total : 34 time to create 1 rle with old method : 0.00012373924255371094 time for calcul the mask position with numpy : 0.00770115852355957 nb_pixel_total : 269 time to create 1 rle with old method : 0.00034308433532714844 time for calcul the mask position with numpy : 0.0066487789154052734 nb_pixel_total : 244 time to create 1 rle with old method : 0.0002994537353515625 time for calcul the mask position with numpy : 0.0066530704498291016 nb_pixel_total : 4 time to create 1 rle with old method : 3.457069396972656e-05 time for calcul the mask position with numpy : 0.0072596073150634766 nb_pixel_total : 21 time to create 1 rle with old method : 5.626678466796875e-05 time for calcul the mask position with numpy : 0.006562471389770508 nb_pixel_total : 340 time to create 1 rle with old method : 0.00041174888610839844 time for calcul the mask position with numpy : 0.006233930587768555 nb_pixel_total : 696 time to create 1 rle with old method : 0.0008318424224853516 time for calcul the mask position with numpy : 0.006098270416259766 nb_pixel_total : 185 time to create 1 rle with old method : 0.0002357959747314453 create new chi : 1.191164493560791 after preparing all the mask , begin to delete the rle from the crop_hashtag_id => VR 28-11-20 : il faut déplacer cela apres le save_crop_hashtag_ids_obj de la ligne 9514 ! we have 0 chi objets contains the rles time to delete rle : 0.0009655952453613281 batch 1 Loaded 82 chid ids of type : 4230 Number RLEs to save : 7833 TO DO : save crop sub photo not yet done ! save time : 1.4643406867980957 map_output_result : {1340531384: (0.0, 'Should be the crop_list due to order', 0.0), 1340531370: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1340531384, 1340531370] Looping around the photos to save general results len do output : 2 /1340531384.Didn't retrieve data . /1340531370.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2627707') ('4234', '20957646', '1340531384', None, None, None, None, None, '2627707') ('4234', None, None, None, None, None, None, None, '2627707') ('4234', '20957646', '1340531370', None, None, None, None, None, '2627707') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 6 time used for this insertion : 0.01353907585144043 save_final save missing photos in datou_result : time spend for datou_step_exec : 8.852011442184448 time spend to save output : 0.013750076293945312 total time spend for step 4 : 8.865761518478394 step5:crop_condition Mon Mar 3 01:21:06 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 4230 Loading chi in step crop for list_pids : 2 ! batch 1 Loaded 168 chid ids of type : 4230 begin to crop the class : papier param for this class : {'min_score': 0.6} filtre for class : papier hashtag_id of this class : 492668766 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 47 About to insert : list_path_to_insert length 47 new photo from crops ! About to upload 47 photos upload in portfolio : 4869462 init cache_photo without model_param we have 47 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740961267_3813386 we have uploaded 47 photos in the portfolio 4869462 time of upload the photos Elapsed time : 17.511168718338013 we have finished the crop for the class : papier begin to crop the class : carton param for this class : {'min_score': 0.6} filtre for class : carton hashtag_id of this class : 492774966 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 15 About to insert : list_path_to_insert length 15 new photo from crops ! About to upload 15 photos upload in portfolio : 4869462 init cache_photo without model_param we have 15 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740961285_3813386 we have uploaded 15 photos in the portfolio 4869462 time of upload the photos Elapsed time : 4.409827947616577 we have finished the crop for the class : carton begin to crop the class : metal param for this class : {'min_score': 0.6} filtre for class : metal hashtag_id of this class : 492628673 begin to crop the class : pet_clair param for this class : {'min_score': 0.6} filtre for class : pet_clair hashtag_id of this class : 2107755846 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! About to upload 9 photos upload in portfolio : 4869462 init cache_photo without model_param we have 9 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740961290_3813386 we have uploaded 9 photos in the portfolio 4869462 time of upload the photos Elapsed time : 9.720218658447266 we have finished the crop for the class : pet_clair begin to crop the class : autre param for this class : {'min_score': 0.6} filtre for class : autre hashtag_id of this class : 494826614 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 29 About to insert : list_path_to_insert length 29 new photo from crops ! About to upload 29 photos upload in portfolio : 4869462 init cache_photo without model_param we have 29 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740961300_3813386 we have uploaded 29 photos in the portfolio 4869462 time of upload the photos Elapsed time : 13.15376615524292 we have finished the crop for the class : autre begin to crop the class : pehd param for this class : {'min_score': 0.6} filtre for class : pehd hashtag_id of this class : 628944319 begin to crop the class : pet_fonce param for this class : {'min_score': 0.6} filtre for class : pet_fonce hashtag_id of this class : 2107755900 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1340531384, 1340531370] Looping around the photos to save general results len do output : 100 /1340556838Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556839Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556840Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556841Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556842Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556843Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556844Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556845Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556846Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556847Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556848Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556849Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556850Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556851Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556852Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556853Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556854Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556855Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556856Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556857Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556858Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556859Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556860Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556861Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556862Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556863Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556864Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556865Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556866Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556867Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556868Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556869Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556870Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556871Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556872Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556873Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556874Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556875Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556876Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556877Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556878Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556879Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556880Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556881Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556882Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556883Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556884Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556885Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556886Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556887Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556888Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556889Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556890Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556891Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556892Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556893Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556894Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556895Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556896Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556897Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556898Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556899Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556900Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556901Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556902Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556903Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556905Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556906Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556907Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556908Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556909Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556915Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556916Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556918Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556919Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556920Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556921Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556922Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556923Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556925Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556926Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556927Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556928Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556929Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556930Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556931Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556932Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556933Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556934Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556935Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556936Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556937Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556938Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556939Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556940Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556941Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556942Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556943Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556944Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340556945Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2627707') ('4234', '20957646', '1340531384', None, None, None, None, None, '2627707') ('4234', None, None, None, None, None, None, None, '2627707') ('4234', '20957646', '1340531370', None, None, None, None, None, '2627707') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 302 time used for this insertion : 0.03610944747924805 save_final save missing photos in datou_result : time spend for datou_step_exec : 46.71013331413269 time spend to save output : 0.039391517639160156 total time spend for step 5 : 46.74952483177185 step6:thcl Mon Mar 3 01:21:53 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! we are using the classfication for only one thcl 3237 time to import caffe and check if the image exist : 0.010605812072753906 time to convert the images to numpy array : 0.02510237693786621 time to import caffe and check if the image exist : 0.004065752029418945 time to convert the images to numpy array : 0.03196835517883301 time to import caffe and check if the image exist : 0.007953643798828125 time to convert the images to numpy array : 0.031178951263427734 time to import caffe and check if the image exist : 0.011763811111450195 time to convert the images to numpy array : 0.02918076515197754 time to import caffe and check if the image exist : 0.006822109222412109 time to convert the images to numpy array : 0.03394889831542969 time to import caffe and check if the image exist : 0.012462377548217773 time to convert the images to numpy array : 0.028944015502929688 time to import caffe and check if the image exist : 0.008872509002685547 time to convert the images to numpy array : 0.033054351806640625 time to import caffe and check if the image exist : 0.009323835372924805 time to convert the images to numpy array : 0.03394913673400879 time to import caffe and check if the image exist : 0.014298200607299805 time to convert the images to numpy array : 0.030173778533935547 time to import caffe and check if the image exist : 0.009286880493164062 time to convert the images to numpy array : 0.03604292869567871 total time to convert the images to numpy array : 0.39179444313049316 list photo_ids error: [] list photo_ids correct : [1340556848, 1340556849, 1340556850, 1340556851, 1340556852, 1340556853, 1340556854, 1340556855, 1340556856, 1340556857, 1340556936, 1340556937, 1340556938, 1340556939, 1340556940, 1340556941, 1340556942, 1340556943, 1340556944, 1340556945, 1340556838, 1340556839, 1340556840, 1340556841, 1340556842, 1340556843, 1340556844, 1340556845, 1340556846, 1340556847, 1340556878, 1340556879, 1340556880, 1340556881, 1340556882, 1340556883, 1340556884, 1340556885, 1340556886, 1340556887, 1340556926, 1340556927, 1340556928, 1340556929, 1340556930, 1340556931, 1340556932, 1340556933, 1340556934, 1340556935, 1340556868, 1340556869, 1340556870, 1340556871, 1340556872, 1340556873, 1340556874, 1340556875, 1340556876, 1340556877, 1340556858, 1340556859, 1340556860, 1340556861, 1340556862, 1340556863, 1340556864, 1340556865, 1340556866, 1340556867, 1340556888, 1340556889, 1340556890, 1340556891, 1340556892, 1340556893, 1340556894, 1340556895, 1340556896, 1340556897, 1340556898, 1340556899, 1340556900, 1340556901, 1340556902, 1340556903, 1340556905, 1340556906, 1340556907, 1340556908, 1340556909, 1340556915, 1340556916, 1340556918, 1340556919, 1340556920, 1340556921, 1340556922, 1340556923, 1340556925] number of photos to traite : 100 try to delete the photos incorrect in DB tagging for thcl : 3237 To do loadFromThcl(), then load ParamDescType : thcl3237 thcls : [{'id': 3237, 'mtr_user_id': 31, 'name': 'learn_rubbia_refus_2500', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton,Film_plastique,PEHD,PET_clair,PET_fonce,Papier,Tetrapak,flou,mal_croppe,metal,refus', 'svm_portfolios_learning': '4865689,4865690,4865686,4865684,4865685,4865688,4865691,4865693,4865692,4865687,4865683', 'photo_hashtag_type': 4158, 'photo_desc_type': 5561, 'type_classification': 'caffe', 'hashtag_id_list': '492774966,2107756122,628944319,2107755846,2107755900,492668766,609991870,492777938,2107755527,492628673,538914404'}] thcl {'id': 3237, 'mtr_user_id': 31, 'name': 'learn_rubbia_refus_2500', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton,Film_plastique,PEHD,PET_clair,PET_fonce,Papier,Tetrapak,flou,mal_croppe,metal,refus', 'svm_portfolios_learning': '4865689,4865690,4865686,4865684,4865685,4865688,4865691,4865693,4865692,4865687,4865683', 'photo_hashtag_type': 4158, 'photo_desc_type': 5561, 'type_classification': 'caffe', 'hashtag_id_list': '492774966,2107756122,628944319,2107755846,2107755900,492668766,609991870,492777938,2107755527,492628673,538914404'} Update svm_hashtag_type_desc : 5561 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5561, 'learn_rubbia_refus_2500', 2048, 2048, 'learn_rubbia_refus_2500', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2021, 12, 2, 19, 10, 8), datetime.datetime(2021, 12, 2, 19, 10, 8)) To loadFromThcl() : net_5561 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 7301 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5561, 'learn_rubbia_refus_2500', 2048, 2048, 'learn_rubbia_refus_2500', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2021, 12, 2, 19, 10, 8), datetime.datetime(2021, 12, 2, 19, 10, 8)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_rubbia_refus_2500 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_rubbia_refus_2500 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_rubbia_refus_2500 /data/models_weight/learn_rubbia_refus_2500/caffemodel size_local : 94358479 size in s3 : 94358479 create time local : 2021-12-03 18:29:39 create time in s3 : 2021-12-02 17:49:16 caffemodel already exist and didn't need to update /data/models_weight/learn_rubbia_refus_2500/deploy.prototxt size_local : 32544 size in s3 : 32544 create time local : 2021-12-03 18:29:39 create time in s3 : 2021-12-02 17:49:15 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_rubbia_refus_2500/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-12-03 18:29:40 create time in s3 : 2021-12-02 18:09:52 mean.npy already exist and didn't need to update /data/models_weight/learn_rubbia_refus_2500/synset_words.txt size_local : 334 size in s3 : 334 create time local : 2021-12-03 18:29:40 create time in s3 : 2021-12-02 18:10:06 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_rubbia_refus_2500/deploy.prototxt caffemodel_filename : /data/models_weight/learn_rubbia_refus_2500/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 6818 max_wait_temp : 1 max_wait : 0 dict_keys(['prob', 'pool5']) time used to do the prepocess of the images : 0.47095537185668945 time used to do the prediction : 0.49017763137817383 save descriptor for thcl : 3237 time to traite the descriptors : 0.5668313503265381 Catched exception ! Connect or reconnect ! storage_type for insertDescriptorsMulti : 3 To insert : 1340556848 To insert : 1340556849 To insert : 1340556850 To insert : 1340556851 To insert : 1340556852 To insert : 1340556853 To insert : 1340556854 To insert : 1340556855 To insert : 1340556856 To insert : 1340556857 To insert : 1340556936 To insert : 1340556937 To insert : 1340556938 To insert : 1340556939 To insert : 1340556940 To insert : 1340556941 To insert : 1340556942 To insert : 1340556943 To insert : 1340556944 To insert : 1340556945 To insert : 1340556838 To insert : 1340556839 To insert : 1340556840 To insert : 1340556841 To insert : 1340556842 To insert : 1340556843 To insert : 1340556844 To insert : 1340556845 To insert : 1340556846 To insert : 1340556847 To insert : 1340556878 To insert : 1340556879 To insert : 1340556880 To insert : 1340556881 To insert : 1340556882 To insert : 1340556883 To insert : 1340556884 To insert : 1340556885 To insert : 1340556886 To insert : 1340556887 To insert : 1340556926 To insert : 1340556927 To insert : 1340556928 To insert : 1340556929 To insert : 1340556930 To insert : 1340556931 To insert : 1340556932 To insert : 1340556933 To insert : 1340556934 To insert : 1340556935 To insert : 1340556868 To insert : 1340556869 To insert : 1340556870 To insert : 1340556871 To insert : 1340556872 To insert : 1340556873 To insert : 1340556874 To insert : 1340556875 To insert : 1340556876 To insert : 1340556877 To insert : 1340556858 To insert : 1340556859 To insert : 1340556860 To insert : 1340556861 To insert : 1340556862 To insert : 1340556863 To insert : 1340556864 To insert : 1340556865 To insert : 1340556866 To insert : 1340556867 To insert : 1340556888 To insert : 1340556889 To insert : 1340556890 To insert : 1340556891 To insert : 1340556892 To insert : 1340556893 To insert : 1340556894 To insert : 1340556895 To insert : 1340556896 To insert : 1340556897 To insert : 1340556898 To insert : 1340556899 To insert : 1340556900 To insert : 1340556901 To insert : 1340556902 To insert : 1340556903 To insert : 1340556905 To insert : 1340556906 To insert : 1340556907 To insert : 1340556908 To insert : 1340556909 To insert : 1340556915 To insert : 1340556916 To insert : 1340556918 To insert : 1340556919 To insert : 1340556920 To insert : 1340556921 To insert : 1340556922 To insert : 1340556923 To insert : 1340556925 time to insert the descriptors : 19.298643112182617 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos SAVE THCL : begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 144 time used for this insertion : 0.018757343292236328 save missing photos in datou_result : time spend for datou_step_exec : 25.272876501083374 time spend to save output : 0.06359505653381348 total time spend for step 6 : 25.336471557617188 step7:ventilate_hashtags_in_portfolio Mon Mar 3 01:22:18 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! Iterating over portfolio : 20957646 get user id for portfolio 20957646 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20957646 AND mptpi.`type`=4230 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('PET_clair','PEHD','PET_fonce','metal','Tetrapak','autre','pehd','environnement','flou','Carton','papier','Papier','mal_croppe','carton','Film_plastique','pet_fonce','refus','pet_clair')) AND mptpi.`min_score`=0.6 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20957646 AND mptpi.`type`=4230 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('PET_clair','PEHD','PET_fonce','metal','Tetrapak','autre','pehd','environnement','flou','Carton','papier','Papier','mal_croppe','carton','Film_plastique','pet_fonce','refus','pet_clair')) AND mptpi.`min_score`=0.6 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20957646 AND mptpi.`type`=4231 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('PET_clair','PEHD','PET_fonce','metal','Tetrapak','autre','pehd','environnement','flou','Carton','papier','Papier','mal_croppe','carton','Film_plastique','pet_fonce','refus','pet_clair')) AND mptpi.`min_score`=0.6 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20957646 AND mptpi.`type`=4231 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('PET_clair','PEHD','PET_fonce','metal','Tetrapak','autre','pehd','environnement','flou','Carton','papier','Papier','mal_croppe','carton','Film_plastique','pet_fonce','refus','pet_clair')) AND mptpi.`min_score`=0.6 To do lien utilise dans velours : https://www.fotonower.com/velours/20958209,20958198,20958207,20958195,20958196,20958197,20958199,20958200,20958205,20958203,20958204,20958206,20958208?tags=pet_clair,pehd,pet_fonce,metal,tetrapak,autre,environnement,flou,carton,papier,mal_croppe,film_plastique,refus&datou_id_consolidate=4235&port_consolidate=20957646 Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1340531384, 1340531370] Looping around the photos to save general results len do output : 1 /20957646. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2627707') ('4234', '20957646', '1340531384', None, None, None, None, None, '2627707') ('4234', None, None, None, None, None, None, None, '2627707') ('4234', '20957646', '1340531370', None, None, None, None, None, '2627707') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 time used for this insertion : 0.012714624404907227 save_final save missing photos in datou_result : time spend for datou_step_exec : 4.110661268234253 time spend to save output : 0.012973546981811523 total time spend for step 7 : 4.1236348152160645 step8:final Mon Mar 3 01:22:22 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1340531384: ('0.20638085550742077',), 1340531370: ('0.20638085550742077',)} new output for save of step final : {1340531384: ('0.20638085550742077',), 1340531370: ('0.20638085550742077',)} [1340531384, 1340531370] Looping around the photos to save general results len do output : 2 /1340531384.Didn't retrieve data . /1340531370.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2627707') ('4234', '20957646', '1340531384', None, None, None, None, None, '2627707') ('4234', None, None, None, None, None, None, None, '2627707') ('4234', '20957646', '1340531370', None, None, None, None, None, '2627707') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 6 time used for this insertion : 0.012588739395141602 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.11414837837219238 time spend to save output : 0.012818336486816406 total time spend for step 8 : 0.1269667148590088 step9:velours_tree Mon Mar 3 01:22:22 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure list_portfolios : 20958209,20958198,20958207,20958195,20958196,20958197,20958199,20958200,20958205,20958203,20958204,20958206,20958208 photo desc type : 5561 - Retrieving photos to tag... query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20958209 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20958198 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20958207 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20958195 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20958196 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20958197 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20958199 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20958200 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20958205 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20958203 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20958204 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20958206 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20958208 ORDER BY ph.size desc - Loading descriptors... Size : 2048 len(descriptors) : 100 Compute structured hierarchical clustering... ward : AgglomerativeClustering(n_clusters=6) ward.labels_ : [3 0 0 1 1 1 0 0 2 5 1 0 5 5 2 1 2 5 2 5 1 2 3 0 3 0 0 1 1 0 0 2 5 1 5 3 1 5 2 1 2 3 5 5 1 2 3 3 3 4 3 0 1 1 2 3 0 1 2 1 1 3 0 1 0 2 2 1 0 0 2 4 4 4 0 2 5 2 5 0 2 2 3 4 3 3 3 4 4 0 2 5 2 5 2 2 3 4 3 3] Elapsed time: 0.014342308044433594 graph_id used : 81075 - Beta version, working pretty good on 11-5-16 ! https://www.fotonower.com/velours/20958209,20958198,20958207,20958195,20958196,20958197,20958199,20958200,20958205,20958203,20958204,20958206,20958208?tags=pet_clair,pehd,pet_fonce,metal,tetrapak,autre,environnement,flou,carton,papier,mal_croppe,film_plastique,refus&datou_id_consolidate=4235&port_consolidate=20957646&tree_id=81075 Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : velours_tree we use saveGeneral [1340531384, 1340531370] Looping around the photos to save general results len do output : 1 /20957646Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2627707') ('4234', '20957646', '1340531384', None, None, None, None, None, '2627707') ('4234', None, None, None, None, None, None, None, '2627707') ('4234', '20957646', '1340531370', None, None, None, None, None, '2627707') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 time used for this insertion : 0.011882781982421875 save_final save missing photos in datou_result : time spend for datou_step_exec : 7.551636695861816 time spend to save output : 0.012311458587646484 total time spend for step 9 : 7.563948154449463 step10:send_mail_cod Mon Mar 3 01:22:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Qualipapia_P20957646_03-03-2025_01_22_30.pdf 20958191 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette209581911740961350 20958180 imagette209581801740961351 20958189 imagette209581891740961351 20958177 imagette209581771740961351 20958178 imagette209581781740961351 20958179 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette209581791740961351 20958182 imagette209581821740961352 20958187 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette209581871740961352 20958185 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette209581851740961352 20958186 imagette209581861740961353 20958188 imagette209581881740961353 20958190 imagette209581901740961353 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20957646 and hashtag_type = 4230 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20958209,20958198,20958207,20958195,20958196,20958197,20958199,20958200,20958205,20958203,20958204,20958206,20958208?tags=pet_clair,pehd,pet_fonce,metal,tetrapak,autre,environnement,flou,carton,papier,mal_croppe,film_plastique,refus&datou_id_consolidate=4235&port_consolidate=20957646&tree_id=81075 your option no_mail is active, we will not send the real mail to your client args[1340531384] : ((1340531384, -7.501900442709465, 492609224), (1340531384, -0.45960958776457533, 496442774), '0.20638085550742077') We are sending mail with results at cod@fotonower.com args[1340531370] : ((1340531370, -7.500526182773834, 492609224), (1340531370, -0.458758268048707, 496442774), '0.20638085550742077') We are sending mail with results at cod@fotonower.com refus_total : 0.20638085550742077 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20957646 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20957646_03-03-2025_01_22_30.pdf results_Qualipapia_P20957646_03-03-2025_01_22_30.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20957646_03-03-2025_01_22_30.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('4234','20957646','results_Qualipapia_P20957646_03-03-2025_01_22_30.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20957646_03-03-2025_01_22_30.pdf','pdf','','0.13','0.20638085550742077') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1340531384, 1340531370] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2627707') ('4234', '20957646', '1340531384', None, None, None, None, None, '2627707') ('4234', None, None, None, None, None, None, None, '2627707') ('4234', '20957646', '1340531370', None, None, None, None, None, '2627707') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 2 time used for this insertion : 0.014489889144897461 save_final save missing photos in datou_result : time spend for datou_step_exec : 4.034560918807983 time spend to save output : 0.014609098434448242 total time spend for step 10 : 4.049170017242432 step11:split_time_score Mon Mar 3 01:22:34 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 3442, 'mtr_user_id': 31, 'name': 'classifieur_2camions_valcor_021122_v1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'deux_camions,camion_droite,camion_gauche,pas_de_camion', 'svm_portfolios_learning': '7659379,7659034,7657685,7657114', 'photo_hashtag_type': 4458, 'photo_desc_type': 5723, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107760533,2107760534,2107760535,2107760536'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('17', 2),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 02032025 20957646 Nombre de photos uploadées : 2 / 23040 (0%) 02032025 20957646 Nombre de photos taguées (types de déchets): 2 / 2 (100%) 02032025 20957646 Nombre de photos taguées (volume) : 0 / 2 (0%) elapsed_time : load_data_split_time_score 1.6689300537109375e-06 elapsed_time : order_list_meta_photo_and_scores 5.4836273193359375e-06 ?? elapsed_time : fill_and_build_computed_from_old_data 0.00022840499877929688 elapsed_time : insert_dashboard_record_day_entry 0.02372884750366211 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957601 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957602 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957603 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957604 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957605 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957606 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957607 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957608 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957609 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957610 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957611 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957612 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957613 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957614 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957615 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957616 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957617 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957618 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957619 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957620 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957621 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957622 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957623 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957624 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957625 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957626 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957627 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957628 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957629 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957630 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957631 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957632 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957633 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957634 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957635 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957636 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957637 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957638 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957639 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957640 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957641 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957642 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957643 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957644 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957645 order by id desc limit 1 Qualite : 0.20638085550742077 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20957646_03-03-2025_01_22_30.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957646 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20957646 AND mptpi.`type`=4230 To do Qualite : 0.22146667136123907 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20957647_03-03-2025_01_16_46.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957647 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20957647 AND mptpi.`type`=4230 To do Qualite : 0.19152548119984242 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20957648_03-03-2025_00_51_08.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957648 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20957648 AND mptpi.`type`=4230 To do Qualite : 0.22877186881093695 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20957649_03-03-2025_00_48_24.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957649 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20957649 AND mptpi.`type`=4230 To do Qualite : 0.23421457348364924 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20957650_03-03-2025_00_52_25.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957650 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20957650 AND mptpi.`type`=4230 To do Qualite : 0.2245714344163915 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20957651_03-03-2025_00_34_24.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957651 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20957651 AND mptpi.`type`=4230 To do Qualite : 0.22212740105907633 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20957652_03-03-2025_00_32_19.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957652 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20957652 AND mptpi.`type`=4230 To do Qualite : 0.21121739094852754 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20957653_03-03-2025_00_18_55.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957653 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20957653 AND mptpi.`type`=4230 To do Qualite : 0.22036621398782735 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20957654_03-03-2025_00_12_07.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957654 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20957654 AND mptpi.`type`=4230 To do Qualite : 0.20524542589930503 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20957655_03-03-2025_00_17_05.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20957655 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20957655 AND mptpi.`type`=4230 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'02032025': {'nb_upload': 2, 'nb_taggue_class': 2, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1340531384, 1340531370] Looping around the photos to save general results len do output : 1 /20957646Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2627707') ('4234', '20957646', '1340531384', None, None, None, None, None, '2627707') ('4234', None, None, None, None, None, None, None, '2627707') ('4234', '20957646', '1340531370', None, None, None, None, None, '2627707') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 time used for this insertion : 0.014104127883911133 save_final save missing photos in datou_result : time spend for datou_step_exec : 35.896703243255615 time spend to save output : 0.014263391494750977 total time spend for step 11 : 35.910966634750366 caffe_path_current : About to save ! 2 After save, about to update current ! ret : 1 len(input) + len(total_photo_id_missing) : 2 set_done_treatment 28.28user 21.00system 2:42.14elapsed 30%CPU (0avgtext+0avgdata 2504448maxresident)k 917296inputs+8280outputs (1095major+1924036minor)pagefaults 0swaps