python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 4234 -l 5 ' -s datou_current_4234 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 1186025 load datou : 4234 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5275, 'learn_RUBBIA_REFUS_AMIENS_23', 16384, 25088, 'learn_RUBBIA_REFUS_AMIENS_23', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 23, 14, 19, 39), datetime.datetime(2021, 4, 23, 14, 19, 39)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5561, 'learn_rubbia_refus_2500', 2048, 2048, 'learn_rubbia_refus_2500', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2021, 12, 2, 19, 10, 8), datetime.datetime(2021, 12, 2, 19, 10, 8)) load thcls load THCL from format json or kwargs add thcl : 2847 in CacheModelConfig load THCL from format json or kwargs add thcl : 3237 in CacheModelConfig load pdts add pdt : 5275 in CacheModelConfig add pdt : 5561 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 list_input_json: [] Current got : datou_id : 4234, datou_cur_ids : ['2597333'] with mtr_portfolio_ids : ['20648165'] and first list_photo_ids : [] new path : /proc/1186025/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, brightness, blur_detection, rle_unique_nms_with_priority, crop_condition, thcl, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , BFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 2 ; length of list_pids : 2 ; length of list_args : 2 time to download the photos : 0.2892117500305176 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Tue Feb 18 21:20:27 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10593 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-18 21:20:30.638635: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-18 21:20:30.663062: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-18 21:20:30.664979: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f0a8c000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-18 21:20:30.665037: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-18 21:20:30.669147: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-18 21:20:30.914324: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2d9f8590 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-18 21:20:30.914392: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-18 21:20:30.915965: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-18 21:20:30.916396: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-18 21:20:30.922922: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-18 21:20:30.925465: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-18 21:20:30.926243: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-18 21:20:30.929160: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-18 21:20:30.930498: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-18 21:20:30.935586: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-18 21:20:30.937196: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-18 21:20:30.937294: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-18 21:20:30.938083: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-18 21:20:30.938100: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-18 21:20:30.938110: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-18 21:20:30.939499: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9815 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-18 21:20:31.209579: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-18 21:20:31.209701: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-18 21:20:31.209723: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-18 21:20:31.209743: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-18 21:20:31.209761: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-18 21:20:31.209779: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-18 21:20:31.209797: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-18 21:20:31.209815: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-18 21:20:31.211361: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-18 21:20:31.212701: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-18 21:20:31.212740: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-18 21:20:31.212759: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-18 21:20:31.212776: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-18 21:20:31.212793: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-18 21:20:31.212810: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-18 21:20:31.212827: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-18 21:20:31.212844: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-18 21:20:31.214362: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-18 21:20:31.214401: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-18 21:20:31.214411: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-18 21:20:31.214421: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-18 21:20:31.216028: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9815 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2847 thcls : [{'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}] thcl {'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5275 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5275, 'learn_RUBBIA_REFUS_AMIENS_23', 16384, 25088, 'learn_RUBBIA_REFUS_AMIENS_23', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 23, 14, 19, 39), datetime.datetime(2021, 4, 23, 14, 19, 39)) {'thcl': {'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'], 'list_hashtags_csv': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'svm_hashtag_type_desc': 5275, 'photo_desc_type': 5275, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_RUBBIA_REFUS_AMIENS_23 NUM_CLASSES 9 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_RUBBIA_REFUS_AMIENS_23 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-18 21:20:38.976651: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-18 21:20:39.177156: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_RUBBIA_REFUS_AMIENS_23 /data/models_weight/learn_RUBBIA_REFUS_AMIENS_23/mask_model.h5 size_local : 256009536 size in s3 : 256009536 create time local : 2021-08-09 09:43:22 create time in s3 : 2021-08-06 18:54:04 mask_model.h5 already exist and didn't need to update list_images length : 2 NEW PHOTO pour l'instant on ne peut pas sauvegarder la photo dans les tile Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -119.14531 max: 149.33828 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 6 time for calcul the mask position with numpy : 8.678436279296875e-05 nb_pixel_total : 882 time to create 1 rle with old method : 0.0016684532165527344 length of segment : 38 time for calcul the mask position with numpy : 6.270408630371094e-05 nb_pixel_total : 808 time to create 1 rle with old method : 0.010605096817016602 length of segment : 29 time for calcul the mask position with numpy : 5.173683166503906e-05 nb_pixel_total : 363 time to create 1 rle with old method : 0.000682830810546875 length of segment : 41 time for calcul the mask position with numpy : 4.4345855712890625e-05 nb_pixel_total : 66 time to create 1 rle with old method : 0.00015282630920410156 length of segment : 29 time for calcul the mask position with numpy : 4.4345855712890625e-05 nb_pixel_total : 258 time to create 1 rle with old method : 0.0005035400390625 length of segment : 37 time for calcul the mask position with numpy : 4.982948303222656e-05 nb_pixel_total : 259 time to create 1 rle with old method : 0.0005025863647460938 length of segment : 40 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -118.39922 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 2 time for calcul the mask position with numpy : 5.6743621826171875e-05 nb_pixel_total : 457 time to create 1 rle with old method : 0.0007927417755126953 length of segment : 34 time for calcul the mask position with numpy : 6.961822509765625e-05 nb_pixel_total : 2551 time to create 1 rle with old method : 0.0031845569610595703 length of segment : 53 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -116.84453 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 4 time for calcul the mask position with numpy : 6.008148193359375e-05 nb_pixel_total : 576 time to create 1 rle with old method : 0.0008485317230224609 length of segment : 23 time for calcul the mask position with numpy : 0.00010251998901367188 nb_pixel_total : 2789 time to create 1 rle with old method : 0.003993988037109375 length of segment : 33 time for calcul the mask position with numpy : 9.751319885253906e-05 nb_pixel_total : 2639 time to create 1 rle with old method : 0.003391742706298828 length of segment : 94 time for calcul the mask position with numpy : 0.0001506805419921875 nb_pixel_total : 2179 time to create 1 rle with old method : 0.002791881561279297 length of segment : 59 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 4 time for calcul the mask position with numpy : 4.839897155761719e-05 nb_pixel_total : 1027 time to create 1 rle with old method : 0.00136566162109375 length of segment : 41 time for calcul the mask position with numpy : 3.6716461181640625e-05 nb_pixel_total : 61 time to create 1 rle with old method : 0.00011801719665527344 length of segment : 12 time for calcul the mask position with numpy : 0.00015735626220703125 nb_pixel_total : 8554 time to create 1 rle with old method : 0.01050710678100586 length of segment : 123 time for calcul the mask position with numpy : 5.221366882324219e-05 nb_pixel_total : 82 time to create 1 rle with old method : 0.0001518726348876953 length of segment : 15 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 251.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 128.38125 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 4 time for calcul the mask position with numpy : 5.054473876953125e-05 nb_pixel_total : 556 time to create 1 rle with old method : 0.0008218288421630859 length of segment : 22 time for calcul the mask position with numpy : 4.172325134277344e-05 nb_pixel_total : 193 time to create 1 rle with old method : 0.0002644062042236328 length of segment : 35 time for calcul the mask position with numpy : 6.628036499023438e-05 nb_pixel_total : 489 time to create 1 rle with old method : 0.0006792545318603516 length of segment : 51 time for calcul the mask position with numpy : 0.0011796951293945312 nb_pixel_total : 40458 time to create 1 rle with old method : 0.0524744987487793 length of segment : 298 Processing 1 images image shape: (400, 400, 3) min: 17.00000 max: 193.00000 molded_images shape: (1, 640, 640, 3) min: -94.23516 max: 72.99062 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 0 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -118.36406 max: 136.09219 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 11 time for calcul the mask position with numpy : 4.673004150390625e-05 nb_pixel_total : 409 time to create 1 rle with old method : 0.0006170272827148438 length of segment : 18 time for calcul the mask position with numpy : 4.291534423828125e-05 nb_pixel_total : 974 time to create 1 rle with old method : 0.0015850067138671875 length of segment : 23 time for calcul the mask position with numpy : 4.506111145019531e-05 nb_pixel_total : 434 time to create 1 rle with old method : 0.0007164478302001953 length of segment : 30 time for calcul the mask position with numpy : 0.00031113624572753906 nb_pixel_total : 18140 time to create 1 rle with old method : 0.020228862762451172 length of segment : 211 time for calcul the mask position with numpy : 9.059906005859375e-05 nb_pixel_total : 1533 time to create 1 rle with old method : 0.0021064281463623047 length of segment : 49 time for calcul the mask position with numpy : 6.914138793945312e-05 nb_pixel_total : 2237 time to create 1 rle with old method : 0.002875804901123047 length of segment : 46 time for calcul the mask position with numpy : 3.504753112792969e-05 nb_pixel_total : 122 time to create 1 rle with old method : 0.00020122528076171875 length of segment : 18 time for calcul the mask position with numpy : 4.00543212890625e-05 nb_pixel_total : 499 time to create 1 rle with old method : 0.0006601810455322266 length of segment : 38 time for calcul the mask position with numpy : 3.600120544433594e-05 nb_pixel_total : 407 time to create 1 rle with old method : 0.0005459785461425781 length of segment : 22 time for calcul the mask position with numpy : 6.079673767089844e-05 nb_pixel_total : 915 time to create 1 rle with old method : 0.0012829303741455078 length of segment : 22 time for calcul the mask position with numpy : 5.459785461425781e-05 nb_pixel_total : 1428 time to create 1 rle with old method : 0.0017578601837158203 length of segment : 51 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -111.93828 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 9 time for calcul the mask position with numpy : 5.91278076171875e-05 nb_pixel_total : 333 time to create 1 rle with old method : 0.00046443939208984375 length of segment : 23 time for calcul the mask position with numpy : 5.245208740234375e-05 nb_pixel_total : 198 time to create 1 rle with old method : 0.00035262107849121094 length of segment : 23 time for calcul the mask position with numpy : 4.458427429199219e-05 nb_pixel_total : 149 time to create 1 rle with old method : 0.0002949237823486328 length of segment : 15 time for calcul the mask position with numpy : 0.000240325927734375 nb_pixel_total : 5348 time to create 1 rle with old method : 0.006632804870605469 length of segment : 95 time for calcul the mask position with numpy : 5.602836608886719e-05 nb_pixel_total : 211 time to create 1 rle with old method : 0.0003535747528076172 length of segment : 17 time for calcul the mask position with numpy : 9.441375732421875e-05 nb_pixel_total : 2658 time to create 1 rle with old method : 0.003360271453857422 length of segment : 62 time for calcul the mask position with numpy : 4.6253204345703125e-05 nb_pixel_total : 50 time to create 1 rle with old method : 0.0001068115234375 length of segment : 8 time for calcul the mask position with numpy : 4.8160552978515625e-05 nb_pixel_total : 748 time to create 1 rle with old method : 0.0009324550628662109 length of segment : 32 time for calcul the mask position with numpy : 4.5299530029296875e-05 nb_pixel_total : 87 time to create 1 rle with old method : 0.00020837783813476562 length of segment : 16 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -121.14141 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 6 time for calcul the mask position with numpy : 4.172325134277344e-05 nb_pixel_total : 301 time to create 1 rle with old method : 0.0005359649658203125 length of segment : 18 time for calcul the mask position with numpy : 7.295608520507812e-05 nb_pixel_total : 635 time to create 1 rle with old method : 0.0008108615875244141 length of segment : 38 time for calcul the mask position with numpy : 7.939338684082031e-05 nb_pixel_total : 1013 time to create 1 rle with old method : 0.0019855499267578125 length of segment : 38 time for calcul the mask position with numpy : 0.000423431396484375 nb_pixel_total : 15237 time to create 1 rle with old method : 0.02070164680480957 length of segment : 229 time for calcul the mask position with numpy : 0.0002963542938232422 nb_pixel_total : 8084 time to create 1 rle with old method : 0.009771108627319336 length of segment : 274 time for calcul the mask position with numpy : 9.417533874511719e-05 nb_pixel_total : 1301 time to create 1 rle with old method : 0.0016279220581054688 length of segment : 41 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 142.06094 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 10 time for calcul the mask position with numpy : 9.179115295410156e-05 nb_pixel_total : 1676 time to create 1 rle with old method : 0.0021708011627197266 length of segment : 147 time for calcul the mask position with numpy : 6.270408630371094e-05 nb_pixel_total : 1813 time to create 1 rle with old method : 0.0023174285888671875 length of segment : 51 time for calcul the mask position with numpy : 8.535385131835938e-05 nb_pixel_total : 560 time to create 1 rle with old method : 0.0007302761077880859 length of segment : 74 time for calcul the mask position with numpy : 4.9114227294921875e-05 nb_pixel_total : 742 time to create 1 rle with old method : 0.0010514259338378906 length of segment : 34 time for calcul the mask position with numpy : 9.179115295410156e-05 nb_pixel_total : 2021 time to create 1 rle with old method : 0.0026426315307617188 length of segment : 156 time for calcul the mask position with numpy : 0.0003173351287841797 nb_pixel_total : 4597 time to create 1 rle with old method : 0.0054378509521484375 length of segment : 227 time for calcul the mask position with numpy : 7.176399230957031e-05 nb_pixel_total : 1521 time to create 1 rle with old method : 0.001924753189086914 length of segment : 79 time for calcul the mask position with numpy : 6.794929504394531e-05 nb_pixel_total : 1761 time to create 1 rle with old method : 0.0022568702697753906 length of segment : 42 time for calcul the mask position with numpy : 5.5789947509765625e-05 nb_pixel_total : 624 time to create 1 rle with old method : 0.0008790493011474609 length of segment : 31 time for calcul the mask position with numpy : 7.05718994140625e-05 nb_pixel_total : 1451 time to create 1 rle with old method : 0.001789093017578125 length of segment : 67 Processing 1 images image shape: (280, 400, 3) min: 10.00000 max: 192.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 68.86953 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 1 time for calcul the mask position with numpy : 0.0009100437164306641 nb_pixel_total : 102513 time to create 1 rle with old method : 0.11539411544799805 length of segment : 282 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 149.26016 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 20 time for calcul the mask position with numpy : 7.367134094238281e-05 nb_pixel_total : 2375 time to create 1 rle with old method : 0.0029261112213134766 length of segment : 66 time for calcul the mask position with numpy : 5.3882598876953125e-05 nb_pixel_total : 821 time to create 1 rle with old method : 0.0010824203491210938 length of segment : 25 time for calcul the mask position with numpy : 3.457069396972656e-05 nb_pixel_total : 331 time to create 1 rle with old method : 0.0004699230194091797 length of segment : 24 time for calcul the mask position with numpy : 3.170967102050781e-05 nb_pixel_total : 174 time to create 1 rle with old method : 0.0002567768096923828 length of segment : 21 time for calcul the mask position with numpy : 7.700920104980469e-05 nb_pixel_total : 3317 time to create 1 rle with old method : 0.0038149356842041016 length of segment : 58 time for calcul the mask position with numpy : 6.747245788574219e-05 nb_pixel_total : 348 time to create 1 rle with old method : 0.0007216930389404297 length of segment : 21 time for calcul the mask position with numpy : 5.53131103515625e-05 nb_pixel_total : 664 time to create 1 rle with old method : 0.0008649826049804688 length of segment : 52 time for calcul the mask position with numpy : 4.506111145019531e-05 nb_pixel_total : 405 time to create 1 rle with old method : 0.0005524158477783203 length of segment : 34 time for calcul the mask position with numpy : 3.838539123535156e-05 nb_pixel_total : 443 time to create 1 rle with old method : 0.0005970001220703125 length of segment : 38 time for calcul the mask position with numpy : 3.3855438232421875e-05 nb_pixel_total : 366 time to create 1 rle with old method : 0.0005087852478027344 length of segment : 19 time for calcul the mask position with numpy : 8.606910705566406e-05 nb_pixel_total : 1995 time to create 1 rle with old method : 0.002519369125366211 length of segment : 99 time for calcul the mask position with numpy : 4.9591064453125e-05 nb_pixel_total : 593 time to create 1 rle with old method : 0.0007317066192626953 length of segment : 46 time for calcul the mask position with numpy : 3.409385681152344e-05 nb_pixel_total : 295 time to create 1 rle with old method : 0.00041604042053222656 length of segment : 16 time for calcul the mask position with numpy : 4.3392181396484375e-05 nb_pixel_total : 900 time to create 1 rle with old method : 0.001255035400390625 length of segment : 25 time for calcul the mask position with numpy : 4.6253204345703125e-05 nb_pixel_total : 592 time to create 1 rle with old method : 0.00079345703125 length of segment : 67 time for calcul the mask position with numpy : 3.528594970703125e-05 nb_pixel_total : 249 time to create 1 rle with old method : 0.0003368854522705078 length of segment : 19 time for calcul the mask position with numpy : 3.0279159545898438e-05 nb_pixel_total : 115 time to create 1 rle with old method : 0.00018334388732910156 length of segment : 17 time for calcul the mask position with numpy : 2.956390380859375e-05 nb_pixel_total : 129 time to create 1 rle with old method : 0.00020933151245117188 length of segment : 9 time for calcul the mask position with numpy : 0.00011610984802246094 nb_pixel_total : 1375 time to create 1 rle with old method : 0.0026216506958007812 length of segment : 55 time for calcul the mask position with numpy : 4.649162292480469e-05 nb_pixel_total : 502 time to create 1 rle with old method : 0.0006191730499267578 length of segment : 43 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 140.97500 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 13 time for calcul the mask position with numpy : 4.00543212890625e-05 nb_pixel_total : 211 time to create 1 rle with old method : 0.0002930164337158203 length of segment : 18 time for calcul the mask position with numpy : 3.4332275390625e-05 nb_pixel_total : 319 time to create 1 rle with old method : 0.00047016143798828125 length of segment : 17 time for calcul the mask position with numpy : 4.2438507080078125e-05 nb_pixel_total : 609 time to create 1 rle with old method : 0.0007605552673339844 length of segment : 32 time for calcul the mask position with numpy : 5.340576171875e-05 nb_pixel_total : 323 time to create 1 rle with old method : 0.00043463706970214844 length of segment : 27 time for calcul the mask position with numpy : 3.0994415283203125e-05 nb_pixel_total : 160 time to create 1 rle with old method : 0.00022935867309570312 length of segment : 15 time for calcul the mask position with numpy : 3.695487976074219e-05 nb_pixel_total : 401 time to create 1 rle with old method : 0.000568389892578125 length of segment : 23 time for calcul the mask position with numpy : 6.222724914550781e-05 nb_pixel_total : 1991 time to create 1 rle with old method : 0.0024747848510742188 length of segment : 64 time for calcul the mask position with numpy : 3.4332275390625e-05 nb_pixel_total : 273 time to create 1 rle with old method : 0.0003829002380371094 length of segment : 21 time for calcul the mask position with numpy : 3.147125244140625e-05 nb_pixel_total : 190 time to create 1 rle with old method : 0.0002834796905517578 length of segment : 19 time for calcul the mask position with numpy : 3.170967102050781e-05 nb_pixel_total : 178 time to create 1 rle with old method : 0.00025534629821777344 length of segment : 17 time for calcul the mask position with numpy : 3.790855407714844e-05 nb_pixel_total : 533 time to create 1 rle with old method : 0.0006756782531738281 length of segment : 35 time for calcul the mask position with numpy : 3.4332275390625e-05 nb_pixel_total : 442 time to create 1 rle with old method : 0.0005679130554199219 length of segment : 26 time for calcul the mask position with numpy : 3.3855438232421875e-05 nb_pixel_total : 472 time to create 1 rle with old method : 0.0005986690521240234 length of segment : 30 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 145.91250 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 15 time for calcul the mask position with numpy : 5.7697296142578125e-05 nb_pixel_total : 259 time to create 1 rle with old method : 0.00037550926208496094 length of segment : 26 time for calcul the mask position with numpy : 3.814697265625e-05 nb_pixel_total : 394 time to create 1 rle with old method : 0.0005753040313720703 length of segment : 26 time for calcul the mask position with numpy : 3.4809112548828125e-05 nb_pixel_total : 381 time to create 1 rle with old method : 0.0005087852478027344 length of segment : 22 time for calcul the mask position with numpy : 3.314018249511719e-05 nb_pixel_total : 291 time to create 1 rle with old method : 0.00041794776916503906 length of segment : 20 time for calcul the mask position with numpy : 3.933906555175781e-05 nb_pixel_total : 526 time to create 1 rle with old method : 0.0007336139678955078 length of segment : 41 time for calcul the mask position with numpy : 3.647804260253906e-05 nb_pixel_total : 339 time to create 1 rle with old method : 0.0004837512969970703 length of segment : 21 time for calcul the mask position with numpy : 0.00011801719665527344 nb_pixel_total : 5903 time to create 1 rle with old method : 0.006747007369995117 length of segment : 110 time for calcul the mask position with numpy : 3.719329833984375e-05 nb_pixel_total : 348 time to create 1 rle with old method : 0.0005009174346923828 length of segment : 21 time for calcul the mask position with numpy : 0.0003674030303955078 nb_pixel_total : 21653 time to create 1 rle with old method : 0.023984909057617188 length of segment : 277 time for calcul the mask position with numpy : 3.266334533691406e-05 nb_pixel_total : 113 time to create 1 rle with old method : 0.0001671314239501953 length of segment : 14 time for calcul the mask position with numpy : 3.2901763916015625e-05 nb_pixel_total : 382 time to create 1 rle with old method : 0.0005087852478027344 length of segment : 23 time for calcul the mask position with numpy : 3.62396240234375e-05 nb_pixel_total : 414 time to create 1 rle with old method : 0.0005466938018798828 length of segment : 34 time for calcul the mask position with numpy : 4.100799560546875e-05 nb_pixel_total : 829 time to create 1 rle with old method : 0.0010788440704345703 length of segment : 40 time for calcul the mask position with numpy : 3.886222839355469e-05 nb_pixel_total : 768 time to create 1 rle with old method : 0.0008833408355712891 length of segment : 44 time for calcul the mask position with numpy : 4.029273986816406e-05 nb_pixel_total : 787 time to create 1 rle with old method : 0.0009529590606689453 length of segment : 45 Processing 1 images image shape: (280, 320, 3) min: 1.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 138.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 1 time for calcul the mask position with numpy : 0.00048828125 nb_pixel_total : 47087 time to create 1 rle with old method : 0.05129718780517578 length of segment : 277 NEW PHOTO pour l'instant on ne peut pas sauvegarder la photo dans les tile Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -118.07109 max: 145.01406 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 5 time for calcul the mask position with numpy : 7.152557373046875e-05 nb_pixel_total : 907 time to create 1 rle with old method : 0.0020072460174560547 length of segment : 42 time for calcul the mask position with numpy : 6.747245788574219e-05 nb_pixel_total : 342 time to create 1 rle with old method : 0.0006871223449707031 length of segment : 32 time for calcul the mask position with numpy : 5.602836608886719e-05 nb_pixel_total : 211 time to create 1 rle with old method : 0.0005104541778564453 length of segment : 40 time for calcul the mask position with numpy : 4.982948303222656e-05 nb_pixel_total : 80 time to create 1 rle with old method : 0.0002498626708984375 length of segment : 28 time for calcul the mask position with numpy : 6.818771362304688e-05 nb_pixel_total : 840 time to create 1 rle with old method : 0.0017387866973876953 length of segment : 39 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -117.00469 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 1 time for calcul the mask position with numpy : 6.747245788574219e-05 nb_pixel_total : 2521 time to create 1 rle with old method : 0.0030524730682373047 length of segment : 54 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -113.18828 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 6 time for calcul the mask position with numpy : 4.9591064453125e-05 nb_pixel_total : 533 time to create 1 rle with old method : 0.0007526874542236328 length of segment : 22 time for calcul the mask position with numpy : 0.00013208389282226562 nb_pixel_total : 2725 time to create 1 rle with old method : 0.0034799575805664062 length of segment : 32 time for calcul the mask position with numpy : 9.846687316894531e-05 nb_pixel_total : 2287 time to create 1 rle with old method : 0.0027730464935302734 length of segment : 78 time for calcul the mask position with numpy : 7.414817810058594e-05 nb_pixel_total : 404 time to create 1 rle with old method : 0.0008289813995361328 length of segment : 27 time for calcul the mask position with numpy : 0.00013017654418945312 nb_pixel_total : 1625 time to create 1 rle with old method : 0.002051830291748047 length of segment : 81 time for calcul the mask position with numpy : 0.0001232624053955078 nb_pixel_total : 2699 time to create 1 rle with old method : 0.003340005874633789 length of segment : 118 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -122.78594 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 4 time for calcul the mask position with numpy : 3.552436828613281e-05 nb_pixel_total : 76 time to create 1 rle with old method : 0.00012683868408203125 length of segment : 13 time for calcul the mask position with numpy : 4.935264587402344e-05 nb_pixel_total : 909 time to create 1 rle with old method : 0.0010833740234375 length of segment : 38 time for calcul the mask position with numpy : 4.029273986816406e-05 nb_pixel_total : 453 time to create 1 rle with old method : 0.0007388591766357422 length of segment : 27 time for calcul the mask position with numpy : 0.00013709068298339844 nb_pixel_total : 8132 time to create 1 rle with old method : 0.009727001190185547 length of segment : 119 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 251.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 125.29141 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 4 time for calcul the mask position with numpy : 4.315376281738281e-05 nb_pixel_total : 612 time to create 1 rle with old method : 0.0008637905120849609 length of segment : 27 time for calcul the mask position with numpy : 9.751319885253906e-05 nb_pixel_total : 2061 time to create 1 rle with old method : 0.002630472183227539 length of segment : 84 time for calcul the mask position with numpy : 3.8623809814453125e-05 nb_pixel_total : 265 time to create 1 rle with old method : 0.0003566741943359375 length of segment : 37 time for calcul the mask position with numpy : 0.0005383491516113281 nb_pixel_total : 44904 time to create 1 rle with old method : 0.04918336868286133 length of segment : 328 Processing 1 images image shape: (400, 400, 3) min: 15.00000 max: 190.00000 molded_images shape: (1, 640, 640, 3) min: -94.73516 max: 70.02187 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 0 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -121.11797 max: 136.92812 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 9 time for calcul the mask position with numpy : 4.673004150390625e-05 nb_pixel_total : 177 time to create 1 rle with old method : 0.0002703666687011719 length of segment : 23 time for calcul the mask position with numpy : 4.5299530029296875e-05 nb_pixel_total : 984 time to create 1 rle with old method : 0.001331329345703125 length of segment : 23 time for calcul the mask position with numpy : 3.24249267578125e-05 nb_pixel_total : 242 time to create 1 rle with old method : 0.0003426074981689453 length of segment : 18 time for calcul the mask position with numpy : 6.651878356933594e-05 nb_pixel_total : 2263 time to create 1 rle with old method : 0.0027582645416259766 length of segment : 49 time for calcul the mask position with numpy : 5.4836273193359375e-05 nb_pixel_total : 1183 time to create 1 rle with old method : 0.0014638900756835938 length of segment : 53 time for calcul the mask position with numpy : 3.719329833984375e-05 nb_pixel_total : 371 time to create 1 rle with old method : 0.0005736351013183594 length of segment : 15 time for calcul the mask position with numpy : 4.363059997558594e-05 nb_pixel_total : 563 time to create 1 rle with old method : 0.0008528232574462891 length of segment : 22 time for calcul the mask position with numpy : 8.487701416015625e-05 nb_pixel_total : 3837 time to create 1 rle with old method : 0.0046422481536865234 length of segment : 72 time for calcul the mask position with numpy : 3.743171691894531e-05 nb_pixel_total : 354 time to create 1 rle with old method : 0.00048613548278808594 length of segment : 20 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -111.76250 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 10 time for calcul the mask position with numpy : 4.076957702636719e-05 nb_pixel_total : 211 time to create 1 rle with old method : 0.00030875205993652344 length of segment : 25 time for calcul the mask position with numpy : 3.504753112792969e-05 nb_pixel_total : 208 time to create 1 rle with old method : 0.00034689903259277344 length of segment : 17 time for calcul the mask position with numpy : 3.457069396972656e-05 nb_pixel_total : 365 time to create 1 rle with old method : 0.000499725341796875 length of segment : 24 time for calcul the mask position with numpy : 2.956390380859375e-05 nb_pixel_total : 71 time to create 1 rle with old method : 0.00014352798461914062 length of segment : 8 time for calcul the mask position with numpy : 3.0994415283203125e-05 nb_pixel_total : 225 time to create 1 rle with old method : 0.0003154277801513672 length of segment : 19 time for calcul the mask position with numpy : 3.981590270996094e-05 nb_pixel_total : 749 time to create 1 rle with old method : 0.0009369850158691406 length of segment : 32 time for calcul the mask position with numpy : 3.0517578125e-05 nb_pixel_total : 186 time to create 1 rle with old method : 0.0002875328063964844 length of segment : 20 time for calcul the mask position with numpy : 2.9802322387695312e-05 nb_pixel_total : 89 time to create 1 rle with old method : 0.0001380443572998047 length of segment : 16 time for calcul the mask position with numpy : 7.82012939453125e-05 nb_pixel_total : 2040 time to create 1 rle with old method : 0.002607583999633789 length of segment : 50 time for calcul the mask position with numpy : 5.3882598876953125e-05 nb_pixel_total : 1407 time to create 1 rle with old method : 0.0018520355224609375 length of segment : 49 Processing 1 images image shape: (400, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -122.77813 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 7 time for calcul the mask position with numpy : 5.221366882324219e-05 nb_pixel_total : 282 time to create 1 rle with old method : 0.0005998611450195312 length of segment : 19 time for calcul the mask position with numpy : 5.650520324707031e-05 nb_pixel_total : 1168 time to create 1 rle with old method : 0.0034170150756835938 length of segment : 39 time for calcul the mask position with numpy : 0.0015947818756103516 nb_pixel_total : 817 time to create 1 rle with old method : 0.001397848129272461 length of segment : 78 time for calcul the mask position with numpy : 5.745887756347656e-05 nb_pixel_total : 972 time to create 1 rle with old method : 0.0026090145111083984 length of segment : 26 time for calcul the mask position with numpy : 9.655952453613281e-05 nb_pixel_total : 3652 time to create 1 rle with old method : 0.007512569427490234 length of segment : 77 time for calcul the mask position with numpy : 0.0006062984466552734 nb_pixel_total : 15591 time to create 1 rle with old method : 0.021620988845825195 length of segment : 217 time for calcul the mask position with numpy : 0.0002512931823730469 nb_pixel_total : 10684 time to create 1 rle with old method : 0.01226496696472168 length of segment : 279 Processing 1 images image shape: (400, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 140.49062 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 8 time for calcul the mask position with numpy : 8.654594421386719e-05 nb_pixel_total : 2687 time to create 1 rle with old method : 0.00323486328125 length of segment : 129 time for calcul the mask position with numpy : 9.369850158691406e-05 nb_pixel_total : 1728 time to create 1 rle with old method : 0.002127408981323242 length of segment : 136 time for calcul the mask position with numpy : 7.104873657226562e-05 nb_pixel_total : 1725 time to create 1 rle with old method : 0.0021321773529052734 length of segment : 84 time for calcul the mask position with numpy : 5.841255187988281e-05 nb_pixel_total : 1813 time to create 1 rle with old method : 0.002257823944091797 length of segment : 42 time for calcul the mask position with numpy : 0.0001862049102783203 nb_pixel_total : 4488 time to create 1 rle with old method : 0.005447864532470703 length of segment : 218 time for calcul the mask position with numpy : 0.0001704692840576172 nb_pixel_total : 3385 time to create 1 rle with old method : 0.0042111873626708984 length of segment : 180 time for calcul the mask position with numpy : 0.0001990795135498047 nb_pixel_total : 3921 time to create 1 rle with old method : 0.004549264907836914 length of segment : 210 time for calcul the mask position with numpy : 5.888938903808594e-05 nb_pixel_total : 1462 time to create 1 rle with old method : 0.0017192363739013672 length of segment : 68 Processing 1 images image shape: (280, 400, 3) min: 9.00000 max: 192.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 63.67422 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 2 time for calcul the mask position with numpy : 0.0009691715240478516 nb_pixel_total : 105923 time to create 1 rle with old method : 0.11730742454528809 length of segment : 283 time for calcul the mask position with numpy : 5.316734313964844e-05 nb_pixel_total : 668 time to create 1 rle with old method : 0.0008051395416259766 length of segment : 54 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 148.83437 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 20 time for calcul the mask position with numpy : 7.295608520507812e-05 nb_pixel_total : 2089 time to create 1 rle with old method : 0.0028047561645507812 length of segment : 45 time for calcul the mask position with numpy : 4.4345855712890625e-05 nb_pixel_total : 865 time to create 1 rle with old method : 0.0011641979217529297 length of segment : 27 time for calcul the mask position with numpy : 8.893013000488281e-05 nb_pixel_total : 3574 time to create 1 rle with old method : 0.0043451786041259766 length of segment : 108 time for calcul the mask position with numpy : 4.076957702636719e-05 nb_pixel_total : 446 time to create 1 rle with old method : 0.0006129741668701172 length of segment : 34 time for calcul the mask position with numpy : 0.00011539459228515625 nb_pixel_total : 1407 time to create 1 rle with old method : 0.0018374919891357422 length of segment : 81 time for calcul the mask position with numpy : 3.218650817871094e-05 nb_pixel_total : 180 time to create 1 rle with old method : 0.00025773048400878906 length of segment : 19 time for calcul the mask position with numpy : 3.0040740966796875e-05 nb_pixel_total : 127 time to create 1 rle with old method : 0.00022864341735839844 length of segment : 8 time for calcul the mask position with numpy : 3.981590270996094e-05 nb_pixel_total : 843 time to create 1 rle with old method : 0.0011453628540039062 length of segment : 26 time for calcul the mask position with numpy : 4.1961669921875e-05 nb_pixel_total : 574 time to create 1 rle with old method : 0.0007376670837402344 length of segment : 48 time for calcul the mask position with numpy : 3.790855407714844e-05 nb_pixel_total : 364 time to create 1 rle with old method : 0.0005123615264892578 length of segment : 25 time for calcul the mask position with numpy : 4.8160552978515625e-05 nb_pixel_total : 536 time to create 1 rle with old method : 0.0007643699645996094 length of segment : 65 time for calcul the mask position with numpy : 3.790855407714844e-05 nb_pixel_total : 360 time to create 1 rle with old method : 0.0005176067352294922 length of segment : 19 time for calcul the mask position with numpy : 3.814697265625e-05 nb_pixel_total : 484 time to create 1 rle with old method : 0.0006225109100341797 length of segment : 40 time for calcul the mask position with numpy : 3.1948089599609375e-05 nb_pixel_total : 245 time to create 1 rle with old method : 0.00034546852111816406 length of segment : 18 time for calcul the mask position with numpy : 7.295608520507812e-05 nb_pixel_total : 3298 time to create 1 rle with old method : 0.003945350646972656 length of segment : 58 time for calcul the mask position with numpy : 3.409385681152344e-05 nb_pixel_total : 311 time to create 1 rle with old method : 0.0004279613494873047 length of segment : 16 time for calcul the mask position with numpy : 3.075599670410156e-05 nb_pixel_total : 146 time to create 1 rle with old method : 0.00022125244140625 length of segment : 15 time for calcul the mask position with numpy : 4.935264587402344e-05 nb_pixel_total : 811 time to create 1 rle with old method : 0.0010001659393310547 length of segment : 50 time for calcul the mask position with numpy : 4.291534423828125e-05 nb_pixel_total : 662 time to create 1 rle with old method : 0.0008940696716308594 length of segment : 41 time for calcul the mask position with numpy : 3.647804260253906e-05 nb_pixel_total : 393 time to create 1 rle with old method : 0.0005519390106201172 length of segment : 22 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 139.17031 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 20 time for calcul the mask position with numpy : 4.553794860839844e-05 nb_pixel_total : 234 time to create 1 rle with old method : 0.0003428459167480469 length of segment : 19 time for calcul the mask position with numpy : 3.552436828613281e-05 nb_pixel_total : 372 time to create 1 rle with old method : 0.0005488395690917969 length of segment : 21 time for calcul the mask position with numpy : 4.00543212890625e-05 nb_pixel_total : 513 time to create 1 rle with old method : 0.0006775856018066406 length of segment : 31 time for calcul the mask position with numpy : 4.220008850097656e-05 nb_pixel_total : 620 time to create 1 rle with old method : 0.0008003711700439453 length of segment : 34 time for calcul the mask position with numpy : 3.409385681152344e-05 nb_pixel_total : 292 time to create 1 rle with old method : 0.0004215240478515625 length of segment : 16 time for calcul the mask position with numpy : 3.0517578125e-05 nb_pixel_total : 98 time to create 1 rle with old method : 0.00018477439880371094 length of segment : 14 time for calcul the mask position with numpy : 3.814697265625e-05 nb_pixel_total : 606 time to create 1 rle with old method : 0.0007727146148681641 length of segment : 30 time for calcul the mask position with numpy : 3.528594970703125e-05 nb_pixel_total : 391 time to create 1 rle with old method : 0.0005328655242919922 length of segment : 21 time for calcul the mask position with numpy : 3.8623809814453125e-05 nb_pixel_total : 533 time to create 1 rle with old method : 0.0006947517395019531 length of segment : 29 time for calcul the mask position with numpy : 6.67572021484375e-05 nb_pixel_total : 2356 time to create 1 rle with old method : 0.0028667449951171875 length of segment : 52 time for calcul the mask position with numpy : 3.0040740966796875e-05 nb_pixel_total : 74 time to create 1 rle with old method : 0.00011730194091796875 length of segment : 15 time for calcul the mask position with numpy : 3.361701965332031e-05 nb_pixel_total : 144 time to create 1 rle with old method : 0.0002338886260986328 length of segment : 22 time for calcul the mask position with numpy : 3.123283386230469e-05 nb_pixel_total : 188 time to create 1 rle with old method : 0.00027489662170410156 length of segment : 19 time for calcul the mask position with numpy : 3.600120544433594e-05 nb_pixel_total : 291 time to create 1 rle with old method : 0.00042176246643066406 length of segment : 17 time for calcul the mask position with numpy : 3.123283386230469e-05 nb_pixel_total : 258 time to create 1 rle with old method : 0.00038623809814453125 length of segment : 16 time for calcul the mask position with numpy : 3.600120544433594e-05 nb_pixel_total : 501 time to create 1 rle with old method : 0.0006308555603027344 length of segment : 34 time for calcul the mask position with numpy : 5.5789947509765625e-05 nb_pixel_total : 2099 time to create 1 rle with old method : 0.0024421215057373047 length of segment : 68 time for calcul the mask position with numpy : 3.743171691894531e-05 nb_pixel_total : 197 time to create 1 rle with old method : 0.00028395652770996094 length of segment : 19 time for calcul the mask position with numpy : 3.5762786865234375e-05 nb_pixel_total : 375 time to create 1 rle with old method : 0.0005164146423339844 length of segment : 23 time for calcul the mask position with numpy : 3.886222839355469e-05 nb_pixel_total : 511 time to create 1 rle with old method : 0.0006401538848876953 length of segment : 37 Processing 1 images image shape: (280, 400, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 145.91250 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 15 time for calcul the mask position with numpy : 4.1484832763671875e-05 nb_pixel_total : 271 time to create 1 rle with old method : 0.0003807544708251953 length of segment : 26 time for calcul the mask position with numpy : 7.82012939453125e-05 nb_pixel_total : 3807 time to create 1 rle with old method : 0.00453948974609375 length of segment : 73 time for calcul the mask position with numpy : 4.100799560546875e-05 nb_pixel_total : 506 time to create 1 rle with old method : 0.0006594657897949219 length of segment : 37 time for calcul the mask position with numpy : 3.337860107421875e-05 nb_pixel_total : 292 time to create 1 rle with old method : 0.00040078163146972656 length of segment : 20 time for calcul the mask position with numpy : 3.3855438232421875e-05 nb_pixel_total : 317 time to create 1 rle with old method : 0.00044465065002441406 length of segment : 21 time for calcul the mask position with numpy : 3.361701965332031e-05 nb_pixel_total : 372 time to create 1 rle with old method : 0.0004849433898925781 length of segment : 21 time for calcul the mask position with numpy : 3.4809112548828125e-05 nb_pixel_total : 415 time to create 1 rle with old method : 0.0005583763122558594 length of segment : 21 time for calcul the mask position with numpy : 3.647804260253906e-05 nb_pixel_total : 385 time to create 1 rle with old method : 0.0005357265472412109 length of segment : 21 time for calcul the mask position with numpy : 5.412101745605469e-05 nb_pixel_total : 118 time to create 1 rle with old method : 0.00025391578674316406 length of segment : 30 time for calcul the mask position with numpy : 3.647804260253906e-05 nb_pixel_total : 656 time to create 1 rle with old method : 0.0008599758148193359 length of segment : 25 time for calcul the mask position with numpy : 4.100799560546875e-05 nb_pixel_total : 546 time to create 1 rle with old method : 0.0006945133209228516 length of segment : 26 time for calcul the mask position with numpy : 0.00027823448181152344 nb_pixel_total : 17780 time to create 1 rle with old method : 0.019700050354003906 length of segment : 219 time for calcul the mask position with numpy : 4.3392181396484375e-05 nb_pixel_total : 264 time to create 1 rle with old method : 0.0003523826599121094 length of segment : 27 time for calcul the mask position with numpy : 3.409385681152344e-05 nb_pixel_total : 303 time to create 1 rle with old method : 0.0004239082336425781 length of segment : 24 time for calcul the mask position with numpy : 3.409385681152344e-05 nb_pixel_total : 333 time to create 1 rle with old method : 0.00046062469482421875 length of segment : 21 Processing 1 images image shape: (280, 320, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 136.78750 image_metas shape: (1, 17) min: 0.00000 max: 640.00000 nb d'objets trouves : 2 time for calcul the mask position with numpy : 7.700920104980469e-05 nb_pixel_total : 66 time to create 1 rle with old method : 0.0001468658447265625 length of segment : 24 time for calcul the mask position with numpy : 0.0005185604095458984 nb_pixel_total : 46366 time to create 1 rle with old method : 0.05225801467895508 length of segment : 277 Detection mask done ! Trying to reset tf kernel 1186647 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 1527 tf kernel not reseted sub process len(results) : 219 len(list_Values) 219 None max_time_sub_proc : 3600 parent process len(results) : 0 len(list_Values) 219 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 6816 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Catched exception ! Connect or reconnect ! Number saved : None batch 1 Loaded 219 chid ids of type : 4228 Number RLEs to save : 12068 save missing photos in datou_result : time spend for datou_step_exec : 24.047654390335083 time spend to save output : 0.909196138381958 total time spend for step 1 : 24.95685052871704 step2:brightness Tue Feb 18 21:20:52 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure inside step calcul brightness treat image : temp/1739910027_1186025_1338370175_e1664205006a149f9441b3c931911147.jpg treat image : temp/1739910027_1186025_1338370174_ff61ca3d0f882776bf5a895e967c0218.jpg Inside saveOutput : final : False verbose : 0 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 2 time used for this insertion : 0.01197671890258789 begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 2 time used for this insertion : 0.011076927185058594 save missing photos in datou_result : time spend for datou_step_exec : 0.8344330787658691 time spend to save output : 0.027994871139526367 total time spend for step 2 : 0.8624279499053955 step3:blur_detection Tue Feb 18 21:20:53 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure inside step blur_detection methode: ratio et variance treat image : temp/1739910027_1186025_1338370175_e1664205006a149f9441b3c931911147.jpg resize: (1080, 1920) 1338370175 -7.624832994914395 treat image : temp/1739910027_1186025_1338370174_ff61ca3d0f882776bf5a895e967c0218.jpg resize: (1080, 1920) 1338370174 -7.592104094296039 Inside saveOutput : final : False verbose : 0 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 2 time used for this insertion : 0.011772871017456055 begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 2 time used for this insertion : 0.010909318923950195 save missing photos in datou_result : time spend for datou_step_exec : 2.170025110244751 time spend to save output : 0.02777385711669922 total time spend for step 3 : 2.19779896736145 step4:rle_unique_nms_with_priority Tue Feb 18 21:20:55 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms batch 1 Loaded 219 chid ids of type : 4228 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++nb_obj : 105 nb_hashtags : 7 time to prepare the origin masks : 1.1402661800384521 time for calcul the mask position with numpy : 0.058804988861083984 nb_pixel_total : 1748258 time to create 1 rle with new method : 0.9107615947723389 time for calcul the mask position with numpy : 0.008785247802734375 nb_pixel_total : 457 time to create 1 rle with old method : 0.0005373954772949219 time for calcul the mask position with numpy : 0.008803844451904297 nb_pixel_total : 882 time to create 1 rle with old method : 0.0010340213775634766 time for calcul the mask position with numpy : 0.008628368377685547 nb_pixel_total : 363 time to create 1 rle with old method : 0.0004439353942871094 time for calcul the mask position with numpy : 0.009002208709716797 nb_pixel_total : 66 time to create 1 rle with old method : 0.00010061264038085938 time for calcul the mask position with numpy : 0.008733510971069336 nb_pixel_total : 36 time to create 1 rle with old method : 7.414817810058594e-05 time for calcul the mask position with numpy : 0.008625984191894531 nb_pixel_total : 22 time to create 1 rle with old method : 7.82012939453125e-05 time for calcul the mask position with numpy : 0.008601665496826172 nb_pixel_total : 2551 time to create 1 rle with old method : 0.0030007362365722656 time for calcul the mask position with numpy : 0.008667707443237305 nb_pixel_total : 9 time to create 1 rle with old method : 6.842613220214844e-05 time for calcul the mask position with numpy : 0.009251117706298828 nb_pixel_total : 40458 time to create 1 rle with old method : 0.044232845306396484 time for calcul the mask position with numpy : 0.008324384689331055 nb_pixel_total : 193 time to create 1 rle with old method : 0.00023102760314941406 time for calcul the mask position with numpy : 0.008453369140625 nb_pixel_total : 576 time to create 1 rle with old method : 0.0006427764892578125 time for calcul the mask position with numpy : 0.008847951889038086 nb_pixel_total : 2789 time to create 1 rle with old method : 0.003135204315185547 time for calcul the mask position with numpy : 0.008428335189819336 nb_pixel_total : 489 time to create 1 rle with old method : 0.0005786418914794922 time for calcul the mask position with numpy : 0.008410215377807617 nb_pixel_total : 170 time to create 1 rle with old method : 0.00023174285888671875 time for calcul the mask position with numpy : 0.008336782455444336 nb_pixel_total : 2639 time to create 1 rle with old method : 0.0030303001403808594 time for calcul the mask position with numpy : 0.00833892822265625 nb_pixel_total : 8554 time to create 1 rle with old method : 0.009184122085571289 time for calcul the mask position with numpy : 0.008208990097045898 nb_pixel_total : 61 time to create 1 rle with old method : 9.202957153320312e-05 time for calcul the mask position with numpy : 0.0083465576171875 nb_pixel_total : 1027 time to create 1 rle with old method : 0.0011096000671386719 time for calcul the mask position with numpy : 0.008380889892578125 nb_pixel_total : 556 time to create 1 rle with old method : 0.0005896091461181641 time for calcul the mask position with numpy : 0.008098125457763672 nb_pixel_total : 82 time to create 1 rle with old method : 0.00010418891906738281 time for calcul the mask position with numpy : 0.008051872253417969 nb_pixel_total : 8084 time to create 1 rle with old method : 0.008762121200561523 time for calcul the mask position with numpy : 0.00842428207397461 nb_pixel_total : 748 time to create 1 rle with old method : 0.0009069442749023438 time for calcul the mask position with numpy : 0.008413076400756836 nb_pixel_total : 635 time to create 1 rle with old method : 0.0006978511810302734 time for calcul the mask position with numpy : 0.008611917495727539 nb_pixel_total : 1013 time to create 1 rle with old method : 0.0012829303741455078 time for calcul the mask position with numpy : 0.0083770751953125 nb_pixel_total : 308 time to create 1 rle with old method : 0.0003616809844970703 time for calcul the mask position with numpy : 0.0083770751953125 nb_pixel_total : 409 time to create 1 rle with old method : 0.00047850608825683594 time for calcul the mask position with numpy : 0.008719205856323242 nb_pixel_total : 85 time to create 1 rle with old method : 0.00014448165893554688 time for calcul the mask position with numpy : 0.008131265640258789 nb_pixel_total : 1813 time to create 1 rle with old method : 0.001981973648071289 time for calcul the mask position with numpy : 0.00827789306640625 nb_pixel_total : 149 time to create 1 rle with old method : 0.00018143653869628906 time for calcul the mask position with numpy : 0.00835728645324707 nb_pixel_total : 301 time to create 1 rle with old method : 0.00035762786865234375 time for calcul the mask position with numpy : 0.008157730102539062 nb_pixel_total : 18020 time to create 1 rle with old method : 0.019483327865600586 time for calcul the mask position with numpy : 0.008303642272949219 nb_pixel_total : 122 time to create 1 rle with old method : 0.0001609325408935547 time for calcul the mask position with numpy : 0.008148431777954102 nb_pixel_total : 5348 time to create 1 rle with old method : 0.006052494049072266 time for calcul the mask position with numpy : 0.008263587951660156 nb_pixel_total : 516 time to create 1 rle with old method : 0.0006988048553466797 time for calcul the mask position with numpy : 0.008194208145141602 nb_pixel_total : 1676 time to create 1 rle with old method : 0.0019488334655761719 time for calcul the mask position with numpy : 0.008233785629272461 nb_pixel_total : 409 time to create 1 rle with old method : 0.0005104541778564453 time for calcul the mask position with numpy : 0.00825047492980957 nb_pixel_total : 560 time to create 1 rle with old method : 0.0006270408630371094 time for calcul the mask position with numpy : 0.008240699768066406 nb_pixel_total : 32 time to create 1 rle with old method : 8.344650268554688e-05 time for calcul the mask position with numpy : 0.007939815521240234 nb_pixel_total : 4597 time to create 1 rle with old method : 0.005274534225463867 time for calcul the mask position with numpy : 0.008350372314453125 nb_pixel_total : 15237 time to create 1 rle with old method : 0.01651167869567871 time for calcul the mask position with numpy : 0.008385658264160156 nb_pixel_total : 75 time to create 1 rle with old method : 0.00011420249938964844 time for calcul the mask position with numpy : 0.008393287658691406 nb_pixel_total : 1475 time to create 1 rle with old method : 0.0016565322875976562 time for calcul the mask position with numpy : 0.008171558380126953 nb_pixel_total : 50 time to create 1 rle with old method : 6.008148193359375e-05 time for calcul the mask position with numpy : 0.00809168815612793 nb_pixel_total : 211 time to create 1 rle with old method : 0.0002892017364501953 time for calcul the mask position with numpy : 0.008336067199707031 nb_pixel_total : 749 time to create 1 rle with old method : 0.0008945465087890625 time for calcul the mask position with numpy : 0.008286714553833008 nb_pixel_total : 333 time to create 1 rle with old method : 0.000377655029296875 time for calcul the mask position with numpy : 0.008271217346191406 nb_pixel_total : 198 time to create 1 rle with old method : 0.00023889541625976562 time for calcul the mask position with numpy : 0.008257389068603516 nb_pixel_total : 407 time to create 1 rle with old method : 0.0004374980926513672 time for calcul the mask position with numpy : 0.008230209350585938 nb_pixel_total : 1533 time to create 1 rle with old method : 0.0017209053039550781 time for calcul the mask position with numpy : 0.008153438568115234 nb_pixel_total : 9 time to create 1 rle with old method : 4.5299530029296875e-05 time for calcul the mask position with numpy : 0.008202791213989258 nb_pixel_total : 1521 time to create 1 rle with old method : 0.0016357898712158203 time for calcul the mask position with numpy : 0.008042335510253906 nb_pixel_total : 31 time to create 1 rle with old method : 6.723403930664062e-05 time for calcul the mask position with numpy : 0.008077383041381836 nb_pixel_total : 434 time to create 1 rle with old method : 0.0004582405090332031 time for calcul the mask position with numpy : 0.008056402206420898 nb_pixel_total : 499 time to create 1 rle with old method : 0.0004985332489013672 time for calcul the mask position with numpy : 0.007993459701538086 nb_pixel_total : 40 time to create 1 rle with old method : 7.939338684082031e-05 time for calcul the mask position with numpy : 0.008080244064331055 nb_pixel_total : 974 time to create 1 rle with old method : 0.001148223876953125 time for calcul the mask position with numpy : 0.008788347244262695 nb_pixel_total : 102513 time to create 1 rle with old method : 0.10622763633728027 time for calcul the mask position with numpy : 0.008237123489379883 nb_pixel_total : 3317 time to create 1 rle with old method : 0.003689289093017578 time for calcul the mask position with numpy : 0.008417129516601562 nb_pixel_total : 443 time to create 1 rle with old method : 0.000514984130859375 time for calcul the mask position with numpy : 0.00842142105102539 nb_pixel_total : 348 time to create 1 rle with old method : 0.00041747093200683594 time for calcul the mask position with numpy : 0.008430242538452148 nb_pixel_total : 405 time to create 1 rle with old method : 0.00046825408935546875 time for calcul the mask position with numpy : 0.008426904678344727 nb_pixel_total : 5 time to create 1 rle with old method : 2.5987625122070312e-05 time for calcul the mask position with numpy : 0.008475542068481445 nb_pixel_total : 366 time to create 1 rle with old method : 0.00063323974609375 time for calcul the mask position with numpy : 0.008487701416015625 nb_pixel_total : 319 time to create 1 rle with old method : 0.00037384033203125 time for calcul the mask position with numpy : 0.008587837219238281 nb_pixel_total : 291 time to create 1 rle with old method : 0.00034999847412109375 time for calcul the mask position with numpy : 0.008527517318725586 nb_pixel_total : 211 time to create 1 rle with old method : 0.0002503395080566406 time for calcul the mask position with numpy : 0.008471250534057617 nb_pixel_total : 48 time to create 1 rle with old method : 7.128715515136719e-05 time for calcul the mask position with numpy : 0.008576631546020508 nb_pixel_total : 15316 time to create 1 rle with old method : 0.01693892478942871 time for calcul the mask position with numpy : 0.00845026969909668 nb_pixel_total : 526 time to create 1 rle with old method : 0.0006165504455566406 time for calcul the mask position with numpy : 0.008299827575683594 nb_pixel_total : 273 time to create 1 rle with old method : 0.0002865791320800781 time for calcul the mask position with numpy : 0.008079290390014648 nb_pixel_total : 331 time to create 1 rle with old method : 0.0003440380096435547 time for calcul the mask position with numpy : 0.008147239685058594 nb_pixel_total : 166 time to create 1 rle with old method : 0.00018835067749023438 time for calcul the mask position with numpy : 0.008256673812866211 nb_pixel_total : 91 time to create 1 rle with old method : 0.0001475811004638672 time for calcul the mask position with numpy : 0.00839996337890625 nb_pixel_total : 190 time to create 1 rle with old method : 0.00023698806762695312 time for calcul the mask position with numpy : 0.008225440979003906 nb_pixel_total : 821 time to create 1 rle with old method : 0.0009732246398925781 time for calcul the mask position with numpy : 0.008410215377807617 nb_pixel_total : 47087 time to create 1 rle with old method : 0.049968719482421875 time for calcul the mask position with numpy : 0.008319377899169922 nb_pixel_total : 664 time to create 1 rle with old method : 0.0007750988006591797 time for calcul the mask position with numpy : 0.008266210556030273 nb_pixel_total : 323 time to create 1 rle with old method : 0.00035381317138671875 time for calcul the mask position with numpy : 0.008285760879516602 nb_pixel_total : 129 time to create 1 rle with old method : 0.00016069412231445312 time for calcul the mask position with numpy : 0.008070707321166992 nb_pixel_total : 2375 time to create 1 rle with old method : 0.0024623870849609375 time for calcul the mask position with numpy : 0.008297920227050781 nb_pixel_total : 592 time to create 1 rle with old method : 0.0006997585296630859 time for calcul the mask position with numpy : 0.008249282836914062 nb_pixel_total : 401 time to create 1 rle with old method : 0.0004146099090576172 time for calcul the mask position with numpy : 0.008407831192016602 nb_pixel_total : 295 time to create 1 rle with old method : 0.00034308433532714844 time for calcul the mask position with numpy : 0.008205175399780273 nb_pixel_total : 653 time to create 1 rle with old method : 0.0008325576782226562 time for calcul the mask position with numpy : 0.008379697799682617 nb_pixel_total : 609 time to create 1 rle with old method : 0.0006816387176513672 time for calcul the mask position with numpy : 0.008465051651000977 nb_pixel_total : 394 time to create 1 rle with old method : 0.0004734992980957031 time for calcul the mask position with numpy : 0.008203268051147461 nb_pixel_total : 1330 time to create 1 rle with old method : 0.0014979839324951172 time for calcul the mask position with numpy : 0.008312463760375977 nb_pixel_total : 1995 time to create 1 rle with old method : 0.0023050308227539062 time for calcul the mask position with numpy : 0.008384943008422852 nb_pixel_total : 6 time to create 1 rle with old method : 2.0265579223632812e-05 time for calcul the mask position with numpy : 0.00829005241394043 nb_pixel_total : 160 time to create 1 rle with old method : 0.0001690387725830078 time for calcul the mask position with numpy : 0.008405447006225586 nb_pixel_total : 22 time to create 1 rle with old method : 3.5762786865234375e-05 time for calcul the mask position with numpy : 0.008175373077392578 nb_pixel_total : 1991 time to create 1 rle with old method : 0.0022382736206054688 time for calcul the mask position with numpy : 0.008274316787719727 nb_pixel_total : 502 time to create 1 rle with old method : 0.0005331039428710938 time for calcul the mask position with numpy : 0.00848388671875 nb_pixel_total : 533 time to create 1 rle with old method : 0.0006654262542724609 time for calcul the mask position with numpy : 0.008694887161254883 nb_pixel_total : 414 time to create 1 rle with old method : 0.0005140304565429688 time for calcul the mask position with numpy : 0.010071039199829102 nb_pixel_total : 14 time to create 1 rle with old method : 4.696846008300781e-05 time for calcul the mask position with numpy : 0.009682416915893555 nb_pixel_total : 5830 time to create 1 rle with old method : 0.0063190460205078125 time for calcul the mask position with numpy : 0.010857582092285156 nb_pixel_total : 9 time to create 1 rle with old method : 4.410743713378906e-05 time for calcul the mask position with numpy : 0.009799003601074219 nb_pixel_total : 27 time to create 1 rle with old method : 8.249282836914062e-05 time for calcul the mask position with numpy : 0.008538961410522461 nb_pixel_total : 768 time to create 1 rle with old method : 0.0009019374847412109 time for calcul the mask position with numpy : 0.008500099182128906 nb_pixel_total : 593 time to create 1 rle with old method : 0.0007212162017822266 time for calcul the mask position with numpy : 0.008535385131835938 nb_pixel_total : 259 time to create 1 rle with old method : 0.0003426074981689453 time for calcul the mask position with numpy : 0.008443832397460938 nb_pixel_total : 86 time to create 1 rle with old method : 0.00014472007751464844 time for calcul the mask position with numpy : 0.008431196212768555 nb_pixel_total : 339 time to create 1 rle with old method : 0.0004050731658935547 time for calcul the mask position with numpy : 0.008375883102416992 nb_pixel_total : 381 time to create 1 rle with old method : 0.0004494190216064453 create new chi : 2.220026969909668 after preparing all the mask , begin to delete the rle from the crop_hashtag_id => VR 28-11-20 : il faut déplacer cela apres le save_crop_hashtag_ids_obj de la ligne 9514 ! we have 0 chi objets contains the rles time to delete rle : 0.00568699836730957 batch 1 Loaded 106 chid ids of type : 4230 Number RLEs to save : 11335 TO DO : save crop sub photo not yet done ! save time : 0.6561989784240723 nb_obj : 113 nb_hashtags : 8 time to prepare the origin masks : 2.138223886489868 time for calcul the mask position with numpy : 0.6591758728027344 nb_pixel_total : 1745839 time to create 1 rle with new method : 0.08963203430175781 time for calcul the mask position with numpy : 0.006482601165771484 nb_pixel_total : 1625 time to create 1 rle with old method : 0.0018393993377685547 time for calcul the mask position with numpy : 0.007227659225463867 nb_pixel_total : 44904 time to create 1 rle with old method : 0.04805612564086914 time for calcul the mask position with numpy : 0.006432771682739258 nb_pixel_total : 80 time to create 1 rle with old method : 0.00010323524475097656 time for calcul the mask position with numpy : 0.006381511688232422 nb_pixel_total : 342 time to create 1 rle with old method : 0.000392913818359375 time for calcul the mask position with numpy : 0.006309032440185547 nb_pixel_total : 907 time to create 1 rle with old method : 0.0010523796081542969 time for calcul the mask position with numpy : 0.0062313079833984375 nb_pixel_total : 34 time to create 1 rle with old method : 6.580352783203125e-05 time for calcul the mask position with numpy : 0.006270170211791992 nb_pixel_total : 22 time to create 1 rle with old method : 6.604194641113281e-05 time for calcul the mask position with numpy : 0.0061762332916259766 nb_pixel_total : 2521 time to create 1 rle with old method : 0.002727508544921875 time for calcul the mask position with numpy : 0.006604433059692383 nb_pixel_total : 101 time to create 1 rle with old method : 0.0001862049102783203 time for calcul the mask position with numpy : 0.006282329559326172 nb_pixel_total : 533 time to create 1 rle with old method : 0.0006289482116699219 time for calcul the mask position with numpy : 0.0062143802642822266 nb_pixel_total : 453 time to create 1 rle with old method : 0.0005395412445068359 time for calcul the mask position with numpy : 0.00974273681640625 nb_pixel_total : 2725 time to create 1 rle with old method : 0.0039136409759521484 time for calcul the mask position with numpy : 0.00665283203125 nb_pixel_total : 504 time to create 1 rle with old method : 0.0006110668182373047 time for calcul the mask position with numpy : 0.0067958831787109375 nb_pixel_total : 2287 time to create 1 rle with old method : 0.002797365188598633 time for calcul the mask position with numpy : 0.00631403923034668 nb_pixel_total : 8132 time to create 1 rle with old method : 0.008803367614746094 time for calcul the mask position with numpy : 0.006501436233520508 nb_pixel_total : 76 time to create 1 rle with old method : 0.00010824203491210938 time for calcul the mask position with numpy : 0.0063707828521728516 nb_pixel_total : 2058 time to create 1 rle with old method : 0.002408742904663086 time for calcul the mask position with numpy : 0.006098031997680664 nb_pixel_total : 265 time to create 1 rle with old method : 0.0003199577331542969 time for calcul the mask position with numpy : 0.006115436553955078 nb_pixel_total : 909 time to create 1 rle with old method : 0.0010004043579101562 time for calcul the mask position with numpy : 0.006395101547241211 nb_pixel_total : 612 time to create 1 rle with old method : 0.0007338523864746094 time for calcul the mask position with numpy : 0.0061337947845458984 nb_pixel_total : 817 time to create 1 rle with old method : 0.0009322166442871094 time for calcul the mask position with numpy : 0.006260395050048828 nb_pixel_total : 1407 time to create 1 rle with old method : 0.0016326904296875 time for calcul the mask position with numpy : 0.007783412933349609 nb_pixel_total : 10684 time to create 1 rle with old method : 0.012607574462890625 time for calcul the mask position with numpy : 0.007162570953369141 nb_pixel_total : 749 time to create 1 rle with old method : 0.001062154769897461 time for calcul the mask position with numpy : 0.006890058517456055 nb_pixel_total : 1168 time to create 1 rle with old method : 0.0013267993927001953 time for calcul the mask position with numpy : 0.008150339126586914 nb_pixel_total : 1813 time to create 1 rle with old method : 0.0020585060119628906 time for calcul the mask position with numpy : 0.006251335144042969 nb_pixel_total : 371 time to create 1 rle with old method : 0.0004963874816894531 time for calcul the mask position with numpy : 0.006346225738525391 nb_pixel_total : 3811 time to create 1 rle with old method : 0.0041522979736328125 time for calcul the mask position with numpy : 0.0061914920806884766 nb_pixel_total : 208 time to create 1 rle with old method : 0.0002703666687011719 time for calcul the mask position with numpy : 0.006167888641357422 nb_pixel_total : 282 time to create 1 rle with old method : 0.0003464221954345703 time for calcul the mask position with numpy : 0.006203174591064453 nb_pixel_total : 177 time to create 1 rle with old method : 0.00021338462829589844 time for calcul the mask position with numpy : 0.006185293197631836 nb_pixel_total : 2687 time to create 1 rle with old method : 0.0029811859130859375 time for calcul the mask position with numpy : 0.006329536437988281 nb_pixel_total : 318 time to create 1 rle with old method : 0.0004153251647949219 time for calcul the mask position with numpy : 0.006184101104736328 nb_pixel_total : 4488 time to create 1 rle with old method : 0.004938602447509766 time for calcul the mask position with numpy : 0.006540060043334961 nb_pixel_total : 134 time to create 1 rle with old method : 0.00027251243591308594 time for calcul the mask position with numpy : 0.007717132568359375 nb_pixel_total : 15591 time to create 1 rle with old method : 0.017674922943115234 time for calcul the mask position with numpy : 0.010050058364868164 nb_pixel_total : 3652 time to create 1 rle with old method : 0.004059314727783203 time for calcul the mask position with numpy : 0.009528398513793945 nb_pixel_total : 119 time to create 1 rle with old method : 0.00021529197692871094 time for calcul the mask position with numpy : 0.008452177047729492 nb_pixel_total : 2040 time to create 1 rle with old method : 0.0023355484008789062 time for calcul the mask position with numpy : 0.009149551391601562 nb_pixel_total : 71 time to create 1 rle with old method : 0.0001323223114013672 time for calcul the mask position with numpy : 0.010164260864257812 nb_pixel_total : 563 time to create 1 rle with old method : 0.0006737709045410156 time for calcul the mask position with numpy : 0.015625476837158203 nb_pixel_total : 186 time to create 1 rle with old method : 0.0004124641418457031 time for calcul the mask position with numpy : 0.010702371597290039 nb_pixel_total : 225 time to create 1 rle with old method : 0.00027561187744140625 time for calcul the mask position with numpy : 0.009916305541992188 nb_pixel_total : 2263 time to create 1 rle with old method : 0.0026650428771972656 time for calcul the mask position with numpy : 0.010392427444458008 nb_pixel_total : 365 time to create 1 rle with old method : 0.0004229545593261719 time for calcul the mask position with numpy : 0.009661674499511719 nb_pixel_total : 211 time to create 1 rle with old method : 0.00026297569274902344 time for calcul the mask position with numpy : 0.009592294692993164 nb_pixel_total : 242 time to create 1 rle with old method : 0.0002868175506591797 time for calcul the mask position with numpy : 0.00949859619140625 nb_pixel_total : 354 time to create 1 rle with old method : 0.0004258155822753906 time for calcul the mask position with numpy : 0.009540319442749023 nb_pixel_total : 1183 time to create 1 rle with old method : 0.0013794898986816406 time for calcul the mask position with numpy : 0.0090789794921875 nb_pixel_total : 1725 time to create 1 rle with old method : 0.0020208358764648438 time for calcul the mask position with numpy : 0.00811004638671875 nb_pixel_total : 89 time to create 1 rle with old method : 0.00011444091796875 time for calcul the mask position with numpy : 0.006205558776855469 nb_pixel_total : 4 time to create 1 rle with old method : 2.5510787963867188e-05 time for calcul the mask position with numpy : 0.006469011306762695 nb_pixel_total : 972 time to create 1 rle with old method : 0.0011272430419921875 time for calcul the mask position with numpy : 0.0065441131591796875 nb_pixel_total : 984 time to create 1 rle with old method : 0.001203775405883789 time for calcul the mask position with numpy : 0.006619453430175781 nb_pixel_total : 811 time to create 1 rle with old method : 0.0009393692016601562 time for calcul the mask position with numpy : 0.007047176361083984 nb_pixel_total : 105923 time to create 1 rle with old method : 0.11983656883239746 time for calcul the mask position with numpy : 0.006438732147216797 nb_pixel_total : 3298 time to create 1 rle with old method : 0.003662109375 time for calcul the mask position with numpy : 0.006362438201904297 nb_pixel_total : 484 time to create 1 rle with old method : 0.0005526542663574219 time for calcul the mask position with numpy : 0.006276369094848633 nb_pixel_total : 364 time to create 1 rle with old method : 0.00044345855712890625 time for calcul the mask position with numpy : 0.006436347961425781 nb_pixel_total : 40 time to create 1 rle with old method : 8.749961853027344e-05 time for calcul the mask position with numpy : 0.00631403923034668 nb_pixel_total : 4 time to create 1 rle with old method : 4.2438507080078125e-05 time for calcul the mask position with numpy : 0.006458282470703125 nb_pixel_total : 360 time to create 1 rle with old method : 0.0004558563232421875 time for calcul the mask position with numpy : 0.0066568851470947266 nb_pixel_total : 292 time to create 1 rle with old method : 0.00037789344787597656 time for calcul the mask position with numpy : 0.006563901901245117 nb_pixel_total : 66 time to create 1 rle with old method : 9.560585021972656e-05 time for calcul the mask position with numpy : 0.006609916687011719 nb_pixel_total : 292 time to create 1 rle with old method : 0.00040841102600097656 time for calcul the mask position with numpy : 0.006584644317626953 nb_pixel_total : 14850 time to create 1 rle with old method : 0.0197446346282959 time for calcul the mask position with numpy : 0.007203578948974609 nb_pixel_total : 234 time to create 1 rle with old method : 0.00045299530029296875 time for calcul the mask position with numpy : 0.0075533390045166016 nb_pixel_total : 1 time to create 1 rle with old method : 2.0503997802734375e-05 time for calcul the mask position with numpy : 0.006855010986328125 nb_pixel_total : 506 time to create 1 rle with old method : 0.0005834102630615234 time for calcul the mask position with numpy : 0.006593942642211914 nb_pixel_total : 144 time to create 1 rle with old method : 0.00018906593322753906 time for calcul the mask position with numpy : 0.006290912628173828 nb_pixel_total : 446 time to create 1 rle with old method : 0.0005192756652832031 time for calcul the mask position with numpy : 0.006688356399536133 nb_pixel_total : 46366 time to create 1 rle with old method : 0.0528409481048584 time for calcul the mask position with numpy : 0.006341218948364258 nb_pixel_total : 80 time to create 1 rle with old method : 0.00011563301086425781 time for calcul the mask position with numpy : 0.0063877105712890625 nb_pixel_total : 291 time to create 1 rle with old method : 0.00035858154296875 time for calcul the mask position with numpy : 0.006316184997558594 nb_pixel_total : 36 time to create 1 rle with old method : 6.318092346191406e-05 time for calcul the mask position with numpy : 0.00615239143371582 nb_pixel_total : 865 time to create 1 rle with old method : 0.0009829998016357422 time for calcul the mask position with numpy : 0.006516456604003906 nb_pixel_total : 4 time to create 1 rle with old method : 2.9087066650390625e-05 time for calcul the mask position with numpy : 0.0065135955810546875 nb_pixel_total : 375 time to create 1 rle with old method : 0.0004475116729736328 time for calcul the mask position with numpy : 0.0063648223876953125 nb_pixel_total : 385 time to create 1 rle with old method : 0.0004475116729736328 time for calcul the mask position with numpy : 0.0062923431396484375 nb_pixel_total : 74 time to create 1 rle with old method : 0.00010085105895996094 time for calcul the mask position with numpy : 0.006299257278442383 nb_pixel_total : 104 time to create 1 rle with old method : 0.00014281272888183594 time for calcul the mask position with numpy : 0.006417512893676758 nb_pixel_total : 1037 time to create 1 rle with old method : 0.0011737346649169922 time for calcul the mask position with numpy : 0.006425142288208008 nb_pixel_total : 546 time to create 1 rle with old method : 0.0006086826324462891 time for calcul the mask position with numpy : 0.0065042972564697266 nb_pixel_total : 2089 time to create 1 rle with old method : 0.005720376968383789 time for calcul the mask position with numpy : 0.0064280033111572266 nb_pixel_total : 417 time to create 1 rle with old method : 0.0005068778991699219 time for calcul the mask position with numpy : 0.006491422653198242 nb_pixel_total : 513 time to create 1 rle with old method : 0.0005903244018554688 time for calcul the mask position with numpy : 0.006731271743774414 nb_pixel_total : 311 time to create 1 rle with old method : 0.0003705024719238281 time for calcul the mask position with numpy : 0.0064046382904052734 nb_pixel_total : 372 time to create 1 rle with old method : 0.0004286766052246094 time for calcul the mask position with numpy : 0.006819725036621094 nb_pixel_total : 787 time to create 1 rle with old method : 0.0009315013885498047 time for calcul the mask position with numpy : 0.006627798080444336 nb_pixel_total : 386 time to create 1 rle with old method : 0.00046706199645996094 time for calcul the mask position with numpy : 0.006590366363525391 nb_pixel_total : 620 time to create 1 rle with old method : 0.0007231235504150391 time for calcul the mask position with numpy : 0.006548881530761719 nb_pixel_total : 3574 time to create 1 rle with old method : 0.004838705062866211 time for calcul the mask position with numpy : 0.0065784454345703125 nb_pixel_total : 317 time to create 1 rle with old method : 0.00041556358337402344 time for calcul the mask position with numpy : 0.006479740142822266 nb_pixel_total : 10 time to create 1 rle with old method : 3.600120544433594e-05 time for calcul the mask position with numpy : 0.0063359737396240234 nb_pixel_total : 188 time to create 1 rle with old method : 0.00022220611572265625 time for calcul the mask position with numpy : 0.006218433380126953 nb_pixel_total : 11 time to create 1 rle with old method : 3.2901763916015625e-05 time for calcul the mask position with numpy : 0.006375789642333984 nb_pixel_total : 2356 time to create 1 rle with old method : 0.002583026885986328 time for calcul the mask position with numpy : 0.006315708160400391 nb_pixel_total : 98 time to create 1 rle with old method : 0.00019311904907226562 time for calcul the mask position with numpy : 0.006264686584472656 nb_pixel_total : 574 time to create 1 rle with old method : 0.0006618499755859375 time for calcul the mask position with numpy : 0.00626373291015625 nb_pixel_total : 23 time to create 1 rle with old method : 5.626678466796875e-05 time for calcul the mask position with numpy : 0.006150484085083008 nb_pixel_total : 501 time to create 1 rle with old method : 0.0005509853363037109 time for calcul the mask position with numpy : 0.006440877914428711 nb_pixel_total : 71 time to create 1 rle with old method : 0.00011467933654785156 time for calcul the mask position with numpy : 0.00632476806640625 nb_pixel_total : 264 time to create 1 rle with old method : 0.0003185272216796875 time for calcul the mask position with numpy : 0.006312131881713867 nb_pixel_total : 656 time to create 1 rle with old method : 0.000766754150390625 time for calcul the mask position with numpy : 0.011902093887329102 nb_pixel_total : 23 time to create 1 rle with old method : 7.963180541992188e-05 time for calcul the mask position with numpy : 0.008484840393066406 nb_pixel_total : 118 time to create 1 rle with old method : 0.00026917457580566406 time for calcul the mask position with numpy : 0.008896827697753906 nb_pixel_total : 3807 time to create 1 rle with old method : 0.0047724246978759766 time for calcul the mask position with numpy : 0.008716106414794922 nb_pixel_total : 415 time to create 1 rle with old method : 0.00046944618225097656 time for calcul the mask position with numpy : 0.008502960205078125 nb_pixel_total : 606 time to create 1 rle with old method : 0.0007109642028808594 time for calcul the mask position with numpy : 0.008461713790893555 nb_pixel_total : 158 time to create 1 rle with old method : 0.000324249267578125 time for calcul the mask position with numpy : 0.008913516998291016 nb_pixel_total : 271 time to create 1 rle with old method : 0.0003273487091064453 time for calcul the mask position with numpy : 0.008492708206176758 nb_pixel_total : 533 time to create 1 rle with old method : 0.0009899139404296875 time for calcul the mask position with numpy : 0.00874948501586914 nb_pixel_total : 371 time to create 1 rle with old method : 0.00043487548828125 create new chi : 1.9547228813171387 after preparing all the mask , begin to delete the rle from the crop_hashtag_id => VR 28-11-20 : il faut déplacer cela apres le save_crop_hashtag_ids_obj de la ligne 9514 ! we have 0 chi objets contains the rles time to delete rle : 0.0013954639434814453 batch 1 Loaded 114 chid ids of type : 4230 Number RLEs to save : 11557 TO DO : save crop sub photo not yet done ! save time : 0.9166808128356934 map_output_result : {1338370175: (0.0, 'Should be the crop_list due to order', 0.0), 1338370174: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1338370175, 1338370174] Looping around the photos to save general results len do output : 2 /1338370175.Didn't retrieve data . /1338370174.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2597333') ('4234', '20648165', '1338370175', None, None, None, None, None, '2597333') ('4234', None, None, None, None, None, None, None, '2597333') ('4234', '20648165', '1338370174', None, None, None, None, None, '2597333') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 6 time used for this insertion : 0.012413263320922852 save_final save missing photos in datou_result : time spend for datou_step_exec : 9.43062949180603 time spend to save output : 0.012668371200561523 total time spend for step 4 : 9.443297863006592 step5:crop_condition Tue Feb 18 21:21:05 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 4230 Loading chi in step crop for list_pids : 2 ! batch 1 Loaded 220 chid ids of type : 4230 begin to crop the class : papier param for this class : {'min_score': 0.6} filtre for class : papier hashtag_id of this class : 492668766 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 79 About to insert : list_path_to_insert length 79 new photo from crops ! About to upload 79 photos upload in portfolio : 4869462 init cache_photo without model_param we have 79 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739910066_1186025 we have uploaded 79 photos in the portfolio 4869462 time of upload the photos Elapsed time : 22.67679238319397 we have finished the crop for the class : papier begin to crop the class : carton param for this class : {'min_score': 0.6} filtre for class : carton hashtag_id of this class : 492774966 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 27 About to insert : list_path_to_insert length 27 new photo from crops ! About to upload 27 photos upload in portfolio : 4869462 init cache_photo without model_param we have 27 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739910090_1186025 we have uploaded 27 photos in the portfolio 4869462 time of upload the photos Elapsed time : 7.341775894165039 we have finished the crop for the class : carton begin to crop the class : metal param for this class : {'min_score': 0.6} filtre for class : metal hashtag_id of this class : 492628673 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739910097_1186025 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.5992052555084229 we have finished the crop for the class : metal begin to crop the class : pet_clair param for this class : {'min_score': 0.6} filtre for class : pet_clair hashtag_id of this class : 2107755846 Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! About to upload 5 photos upload in portfolio : 4869462 init cache_photo without model_param we have 5 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739910098_1186025 we have uploaded 5 photos in the portfolio 4869462 time of upload the photos Elapsed time : 3.472395896911621 we have finished the crop for the class : pet_clair begin to crop the class : autre param for this class : {'min_score': 0.6} filtre for class : autre hashtag_id of this class : 494826614 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 27 About to insert : list_path_to_insert length 27 new photo from crops ! About to upload 27 photos upload in portfolio : 4869462 init cache_photo without model_param we have 27 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739910102_1186025 we have uploaded 27 photos in the portfolio 4869462 time of upload the photos Elapsed time : 7.483609914779663 we have finished the crop for the class : autre begin to crop the class : pehd param for this class : {'min_score': 0.6} filtre for class : pehd hashtag_id of this class : 628944319 begin to crop the class : pet_fonce param for this class : {'min_score': 0.6} filtre for class : pet_fonce hashtag_id of this class : 2107755900 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1338370175, 1338370174] Looping around the photos to save general results len do output : 139 /1338388580Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388581Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388582Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388583Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388584Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388585Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388586Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388587Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388588Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388589Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388590Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388591Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388592Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388593Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388594Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388595Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388596Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388597Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388598Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388599Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388600Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388601Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388602Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388603Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388604Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388605Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388606Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388607Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388608Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388609Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388610Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388611Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388612Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388613Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388614Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388615Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388616Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388617Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388618Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388619Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388620Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388621Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388622Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388623Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388625Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388626Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388627Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388629Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388630Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388631Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388632Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388633Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388634Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388636Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388637Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388638Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388639Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388640Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388641Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388642Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388643Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388644Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388645Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388646Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388647Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388648Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388649Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388650Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388651Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388652Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388653Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388654Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388655Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388656Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388657Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388658Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388659Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388660Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388661Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388664Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388665Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388666Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388667Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388668Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388669Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388670Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388671Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388672Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388673Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388674Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388675Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388676Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388677Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388678Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388679Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388680Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388681Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388682Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388683Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388684Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388685Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388686Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388687Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388689Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388690Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388691Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388693Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388694Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388695Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388696Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388697Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388698Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388703Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388704Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388705Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388706Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388707Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388708Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388709Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388710Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388711Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388712Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388713Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388714Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388715Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388716Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388717Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388718Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388719Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388720Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388721Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388722Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388723Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388724Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388725Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388726Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388727Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388728Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1338388729Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2597333') ('4234', '20648165', '1338370175', None, None, None, None, None, '2597333') ('4234', None, None, None, None, None, None, None, '2597333') ('4234', '20648165', '1338370174', None, None, None, None, None, '2597333') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 419 time used for this insertion : 0.03292250633239746 save_final save missing photos in datou_result : time spend for datou_step_exec : 44.18376064300537 time spend to save output : 0.038283348083496094 total time spend for step 5 : 44.22204399108887 step6:thcl Tue Feb 18 21:21:49 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! we are using the classfication for only one thcl 3237 time to import caffe and check if the image exist : 0.007782697677612305 time to convert the images to numpy array : 0.041646718978881836 time to import caffe and check if the image exist : 0.007742166519165039 time to convert the images to numpy array : 0.04253411293029785 time to import caffe and check if the image exist : 0.009478569030761719 time to convert the images to numpy array : 0.04226851463317871 time to import caffe and check if the image exist : 0.007108211517333984 time to convert the images to numpy array : 0.04938030242919922 time to import caffe and check if the image exist : 0.010398149490356445 time to convert the images to numpy array : 0.04675126075744629 time to import caffe and check if the image exist : 0.015537500381469727 time to convert the images to numpy array : 0.041169166564941406 time to import caffe and check if the image exist : 0.014936685562133789 time to convert the images to numpy array : 0.04641890525817871 time to import caffe and check if the image exist : 0.014507055282592773 time to convert the images to numpy array : 0.046944618225097656 time to import caffe and check if the image exist : 0.014619588851928711 time to convert the images to numpy array : 0.04822230339050293 time to import caffe and check if the image exist : 0.014875173568725586 time to convert the images to numpy array : 0.0473480224609375 total time to convert the images to numpy array : 0.3231213092803955 list photo_ids error: [] list photo_ids correct : [1338388639, 1338388640, 1338388641, 1338388642, 1338388643, 1338388644, 1338388645, 1338388646, 1338388647, 1338388648, 1338388649, 1338388650, 1338388651, 1338388652, 1338388622, 1338388623, 1338388625, 1338388626, 1338388627, 1338388629, 1338388630, 1338388631, 1338388632, 1338388633, 1338388634, 1338388636, 1338388637, 1338388638, 1338388594, 1338388595, 1338388596, 1338388597, 1338388598, 1338388599, 1338388600, 1338388601, 1338388602, 1338388603, 1338388604, 1338388605, 1338388606, 1338388607, 1338388580, 1338388581, 1338388582, 1338388583, 1338388584, 1338388585, 1338388586, 1338388587, 1338388588, 1338388589, 1338388590, 1338388591, 1338388592, 1338388593, 1338388669, 1338388670, 1338388671, 1338388672, 1338388673, 1338388674, 1338388675, 1338388676, 1338388677, 1338388678, 1338388679, 1338388680, 1338388681, 1338388682, 1338388717, 1338388718, 1338388719, 1338388720, 1338388721, 1338388722, 1338388723, 1338388724, 1338388725, 1338388726, 1338388727, 1338388728, 1338388729, 1338388653, 1338388654, 1338388655, 1338388656, 1338388657, 1338388658, 1338388659, 1338388660, 1338388661, 1338388664, 1338388665, 1338388666, 1338388667, 1338388668, 1338388703, 1338388704, 1338388705, 1338388706, 1338388707, 1338388708, 1338388709, 1338388710, 1338388711, 1338388712, 1338388713, 1338388714, 1338388715, 1338388716, 1338388608, 1338388609, 1338388610, 1338388611, 1338388612, 1338388613, 1338388614, 1338388615, 1338388616, 1338388617, 1338388618, 1338388619, 1338388620, 1338388621, 1338388683, 1338388684, 1338388685, 1338388686, 1338388687, 1338388689, 1338388690, 1338388691, 1338388693, 1338388694, 1338388695, 1338388696, 1338388697, 1338388698] number of photos to traite : 139 try to delete the photos incorrect in DB tagging for thcl : 3237 To do loadFromThcl(), then load ParamDescType : thcl3237 thcls : [{'id': 3237, 'mtr_user_id': 31, 'name': 'learn_rubbia_refus_2500', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton,Film_plastique,PEHD,PET_clair,PET_fonce,Papier,Tetrapak,flou,mal_croppe,metal,refus', 'svm_portfolios_learning': '4865689,4865690,4865686,4865684,4865685,4865688,4865691,4865693,4865692,4865687,4865683', 'photo_hashtag_type': 4158, 'photo_desc_type': 5561, 'type_classification': 'caffe', 'hashtag_id_list': '492774966,2107756122,628944319,2107755846,2107755900,492668766,609991870,492777938,2107755527,492628673,538914404'}] thcl {'id': 3237, 'mtr_user_id': 31, 'name': 'learn_rubbia_refus_2500', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton,Film_plastique,PEHD,PET_clair,PET_fonce,Papier,Tetrapak,flou,mal_croppe,metal,refus', 'svm_portfolios_learning': '4865689,4865690,4865686,4865684,4865685,4865688,4865691,4865693,4865692,4865687,4865683', 'photo_hashtag_type': 4158, 'photo_desc_type': 5561, 'type_classification': 'caffe', 'hashtag_id_list': '492774966,2107756122,628944319,2107755846,2107755900,492668766,609991870,492777938,2107755527,492628673,538914404'} Update svm_hashtag_type_desc : 5561 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5561, 'learn_rubbia_refus_2500', 2048, 2048, 'learn_rubbia_refus_2500', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2021, 12, 2, 19, 10, 8), datetime.datetime(2021, 12, 2, 19, 10, 8)) To loadFromThcl() : net_5561 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10593 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5561, 'learn_rubbia_refus_2500', 2048, 2048, 'learn_rubbia_refus_2500', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2021, 12, 2, 19, 10, 8), datetime.datetime(2021, 12, 2, 19, 10, 8)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_rubbia_refus_2500 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_rubbia_refus_2500 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_rubbia_refus_2500 /data/models_weight/learn_rubbia_refus_2500/caffemodel size_local : 94358479 size in s3 : 94358479 create time local : 2021-12-03 18:29:39 create time in s3 : 2021-12-02 17:49:16 caffemodel already exist and didn't need to update /data/models_weight/learn_rubbia_refus_2500/deploy.prototxt size_local : 32544 size in s3 : 32544 create time local : 2021-12-03 18:29:39 create time in s3 : 2021-12-02 17:49:15 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_rubbia_refus_2500/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-12-03 18:29:40 create time in s3 : 2021-12-02 18:09:52 mean.npy already exist and didn't need to update /data/models_weight/learn_rubbia_refus_2500/synset_words.txt size_local : 334 size in s3 : 334 create time local : 2021-12-03 18:29:40 create time in s3 : 2021-12-02 18:10:06 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_rubbia_refus_2500/deploy.prototxt caffemodel_filename : /data/models_weight/learn_rubbia_refus_2500/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10374 max_wait_temp : 1 max_wait : 0 dict_keys(['pool5', 'prob']) time used to do the prepocess of the images : 0.6276848316192627 time used to do the prediction : 0.5808584690093994 save descriptor for thcl : 3237 time to traite the descriptors : 0.7722115516662598 Catched exception ! Connect or reconnect ! storage_type for insertDescriptorsMulti : 3 To insert : 1338388639 To insert : 1338388640 To insert : 1338388641 To insert : 1338388642 To insert : 1338388643 To insert : 1338388644 To insert : 1338388645 To insert : 1338388646 To insert : 1338388647 To insert : 1338388648 To insert : 1338388649 To insert : 1338388650 To insert : 1338388651 To insert : 1338388652 To insert : 1338388622 To insert : 1338388623 To insert : 1338388625 To insert : 1338388626 To insert : 1338388627 To insert : 1338388629 To insert : 1338388630 To insert : 1338388631 To insert : 1338388632 To insert : 1338388633 To insert : 1338388634 To insert : 1338388636 To insert : 1338388637 To insert : 1338388638 To insert : 1338388594 To insert : 1338388595 To insert : 1338388596 To insert : 1338388597 To insert : 1338388598 To insert : 1338388599 To insert : 1338388600 To insert : 1338388601 To insert : 1338388602 To insert : 1338388603 To insert : 1338388604 To insert : 1338388605 To insert : 1338388606 To insert : 1338388607 To insert : 1338388580 To insert : 1338388581 To insert : 1338388582 To insert : 1338388583 To insert : 1338388584 To insert : 1338388585 To insert : 1338388586 To insert : 1338388587 To insert : 1338388588 To insert : 1338388589 To insert : 1338388590 To insert : 1338388591 To insert : 1338388592 To insert : 1338388593 To insert : 1338388669 To insert : 1338388670 To insert : 1338388671 To insert : 1338388672 To insert : 1338388673 To insert : 1338388674 To insert : 1338388675 To insert : 1338388676 To insert : 1338388677 To insert : 1338388678 To insert : 1338388679 To insert : 1338388680 To insert : 1338388681 To insert : 1338388682 To insert : 1338388717 To insert : 1338388718 To insert : 1338388719 To insert : 1338388720 To insert : 1338388721 To insert : 1338388722 To insert : 1338388723 To insert : 1338388724 To insert : 1338388725 To insert : 1338388726 To insert : 1338388727 To insert : 1338388728 To insert : 1338388729 To insert : 1338388653 To insert : 1338388654 To insert : 1338388655 To insert : 1338388656 To insert : 1338388657 To insert : 1338388658 To insert : 1338388659 To insert : 1338388660 To insert : 1338388661 To insert : 1338388664 To insert : 1338388665 To insert : 1338388666 To insert : 1338388667 To insert : 1338388668 To insert : 1338388703 To insert : 1338388704 To insert : 1338388705 To insert : 1338388706 To insert : 1338388707 To insert : 1338388708 To insert : 1338388709 To insert : 1338388710 To insert : 1338388711 To insert : 1338388712 To insert : 1338388713 To insert : 1338388714 To insert : 1338388715 To insert : 1338388716 To insert : 1338388608 To insert : 1338388609 To insert : 1338388610 To insert : 1338388611 To insert : 1338388612 To insert : 1338388613 To insert : 1338388614 To insert : 1338388615 To insert : 1338388616 To insert : 1338388617 To insert : 1338388618 To insert : 1338388619 To insert : 1338388620 To insert : 1338388621 To insert : 1338388683 To insert : 1338388684 To insert : 1338388685 To insert : 1338388686 To insert : 1338388687 To insert : 1338388689 To insert : 1338388690 To insert : 1338388691 To insert : 1338388693 To insert : 1338388694 To insert : 1338388695 To insert : 1338388696 To insert : 1338388697 To insert : 1338388698 time to insert the descriptors : 26.701093196868896 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos SAVE THCL : begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 226 time used for this insertion : 0.022138118743896484 save missing photos in datou_result : time spend for datou_step_exec : 32.95631170272827 time spend to save output : 0.06898617744445801 total time spend for step 6 : 33.02529788017273 step7:ventilate_hashtags_in_portfolio Tue Feb 18 21:22:22 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! Iterating over portfolio : 20648165 get user id for portfolio 20648165 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20648165 AND mptpi.`type`=4230 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('pehd','papier','flou','PEHD','carton','metal','Film_plastique','pet_clair','autre','pet_fonce','refus','Carton','PET_clair','Tetrapak','PET_fonce','Papier','mal_croppe','environnement')) AND mptpi.`min_score`=0.6 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20648165 AND mptpi.`type`=4230 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('pehd','papier','flou','PEHD','carton','metal','Film_plastique','pet_clair','autre','pet_fonce','refus','Carton','PET_clair','Tetrapak','PET_fonce','Papier','mal_croppe','environnement')) AND mptpi.`min_score`=0.6 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20648165 AND mptpi.`type`=4231 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('pehd','papier','flou','PEHD','carton','metal','Film_plastique','pet_clair','autre','pet_fonce','refus','Carton','PET_clair','Tetrapak','PET_fonce','Papier','mal_croppe','environnement')) AND mptpi.`min_score`=0.6 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20648165 AND mptpi.`type`=4231 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('pehd','papier','flou','PEHD','carton','metal','Film_plastique','pet_clair','autre','pet_fonce','refus','Carton','PET_clair','Tetrapak','PET_fonce','Papier','mal_croppe','environnement')) AND mptpi.`min_score`=0.6 To do lien utilise dans velours : https://www.fotonower.com/velours/20654538,20654550,20654537,20654546,20654540,20654541,20654547,20654543,20654549,20654545,20654548,20654551,20654552?tags=pehd,papier,flou,carton,metal,film_plastique,pet_clair,autre,pet_fonce,refus,tetrapak,mal_croppe,environnement&datou_id_consolidate=4235&port_consolidate=20648165 Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1338370175, 1338370174] Looping around the photos to save general results len do output : 1 /20648165. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2597333') ('4234', '20648165', '1338370175', None, None, None, None, None, '2597333') ('4234', None, None, None, None, None, None, None, '2597333') ('4234', '20648165', '1338370174', None, None, None, None, None, '2597333') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 time used for this insertion : 0.31066060066223145 save_final save missing photos in datou_result : time spend for datou_step_exec : 2.474489450454712 time spend to save output : 0.31093311309814453 total time spend for step 7 : 2.7854225635528564 step8:final Tue Feb 18 21:22:25 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1338370175: ('0.27055926134380387',), 1338370174: ('0.27055926134380387',)} new output for save of step final : {1338370175: ('0.27055926134380387',), 1338370174: ('0.27055926134380387',)} [1338370175, 1338370174] Looping around the photos to save general results len do output : 2 /1338370175.Didn't retrieve data . /1338370174.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2597333') ('4234', '20648165', '1338370175', None, None, None, None, None, '2597333') ('4234', None, None, None, None, None, None, None, '2597333') ('4234', '20648165', '1338370174', None, None, None, None, None, '2597333') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 6 time used for this insertion : 0.01258540153503418 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.11819267272949219 time spend to save output : 0.01281428337097168 total time spend for step 8 : 0.13100695610046387 step9:velours_tree Tue Feb 18 21:22:25 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure list_portfolios : 20654538,20654550,20654537,20654546,20654540,20654541,20654547,20654543,20654549,20654545,20654548,20654551,20654552 photo desc type : 5561 - Retrieving photos to tag... query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20654538 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20654550 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20654537 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20654546 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20654540 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20654541 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20654547 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20654543 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20654549 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20654545 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20654548 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20654551 ORDER BY ph.size desc query : SELECT ph.photo_id FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20654552 ORDER BY ph.size desc - Loading descriptors... Size : 2048 len(descriptors) : 139 Compute structured hierarchical clustering... ward : AgglomerativeClustering(n_clusters=6) ward.labels_ : [4 0 0 1 0 2 3 4 2 2 4 5 1 2 1 4 4 4 2 5 2 5 2 2 3 2 3 2 4 4 0 2 0 2 4 2 2 4 4 4 2 4 0 1 0 3 2 0 2 3 2 4 2 2 2 1 2 4 4 2 2 2 4 2 4 4 4 0 4 2 2 4 4 2 4 2 4 3 4 1 0 4 1 2 1 1 5 2 3 4 4 3 1 0 1 1 5 1 4 5 5 3 1 4 4 3 2 2 0 0 2 0 3 4 1 3 2 2 3 3 2 4 4 2 3 3 2 3 2 3 3 3 1 2 4 3 4 2 3] Elapsed time: 0.020198822021484375 graph_id used : 79252 - Beta version, working pretty good on 11-5-16 ! https://www.fotonower.com/velours/20654538,20654550,20654537,20654546,20654540,20654541,20654547,20654543,20654549,20654545,20654548,20654551,20654552?tags=pehd,papier,flou,carton,metal,film_plastique,pet_clair,autre,pet_fonce,refus,tetrapak,mal_croppe,environnement&datou_id_consolidate=4235&port_consolidate=20648165&tree_id=79252 Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : velours_tree we use saveGeneral [1338370175, 1338370174] Looping around the photos to save general results len do output : 1 /20648165Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2597333') ('4234', '20648165', '1338370175', None, None, None, None, None, '2597333') ('4234', None, None, None, None, None, None, None, '2597333') ('4234', '20648165', '1338370174', None, None, None, None, None, '2597333') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 time used for this insertion : 0.01163482666015625 save_final save missing photos in datou_result : time spend for datou_step_exec : 13.344928979873657 time spend to save output : 0.011837482452392578 total time spend for step 9 : 13.35676646232605 step10:send_mail_cod Tue Feb 18 21:22:38 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Qualipapia_P20648165_18-02-2025_21_22_38.pdf 20654520 imagette206545201739910158 20654532 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette206545321739910158 20654519 imagette206545191739910159 20654528 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette206545281739910159 20654522 change filename to text .imagette206545221739910160 20654523 imagette206545231739910160 20654529 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette206545291739910160 20654525 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette206545251739910160 20654531 imagette206545311739910161 20654527 imagette206545271739910161 20654530 imagette206545301739910161 20654533 imagette206545331739910161 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20648165 and hashtag_type = 4230 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20654538,20654550,20654537,20654546,20654540,20654541,20654547,20654543,20654549,20654545,20654548,20654551,20654552?tags=pehd,papier,flou,carton,metal,film_plastique,pet_clair,autre,pet_fonce,refus,tetrapak,mal_croppe,environnement&datou_id_consolidate=4235&port_consolidate=20648165&tree_id=79252 your option no_mail is active, we will not send the real mail to your client args[1338370175] : ((1338370175, -7.624832994914395, 492609224), (1338370175, -0.5799435859536028, 501862349), '0.27055926134380387') We are sending mail with results at cod@fotonower.com args[1338370174] : ((1338370174, -7.592104094296039, 492609224), (1338370174, -0.6301784569226283, 501862349), '0.27055926134380387') We are sending mail with results at cod@fotonower.com refus_total : 0.27055926134380387 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20648165 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 error for generate pdf : 'flou' Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1338370175, 1338370174] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2597333') ('4234', '20648165', '1338370175', None, None, None, None, None, '2597333') ('4234', None, None, None, None, None, None, None, '2597333') ('4234', '20648165', '1338370174', None, None, None, None, None, '2597333') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 2 time used for this insertion : 0.013139009475708008 save_final save missing photos in datou_result : time spend for datou_step_exec : 3.076125144958496 time spend to save output : 0.013245105743408203 total time spend for step 10 : 3.0893702507019043 step11:split_time_score Tue Feb 18 21:22:41 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 3442, 'mtr_user_id': 31, 'name': 'classifieur_2camions_valcor_021122_v1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'deux_camions,camion_droite,camion_gauche,pas_de_camion', 'svm_portfolios_learning': '7659379,7659034,7657685,7657114', 'photo_hashtag_type': 4458, 'photo_desc_type': 5723, 'type_classification': 'tf_classification2', 'hashtag_id_list': '2107760533,2107760534,2107760535,2107760536'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('17', 2),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 18022025 20648165 Nombre de photos uploadées : 2 / 23040 (0%) 18022025 20648165 Nombre de photos taguées (types de déchets): 2 / 2 (100%) 18022025 20648165 Nombre de photos taguées (volume) : 0 / 2 (0%) elapsed_time : load_data_split_time_score 2.6226043701171875e-06 elapsed_time : order_list_meta_photo_and_scores 8.106231689453125e-06 ?? elapsed_time : fill_and_build_computed_from_old_data 0.0002262592315673828 elapsed_time : insert_dashboard_record_day_entry 0.02388906478881836 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.31770078358789855 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20635987_18-02-2025_12_33_38.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20635987 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20635987 AND mptpi.`type`=4230 To do Qualite : 0.27942350974274305 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20635989_18-02-2025_12_06_30.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20635989 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20635989 AND mptpi.`type`=4230 To do Qualite : 0.24472260075892122 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20635991_18-02-2025_12_18_20.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20635991 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20635991 AND mptpi.`type`=4230 To do Qualite : 0.2520744446701883 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20637016_18-02-2025_13_17_29.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20637016 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20637016 AND mptpi.`type`=4230 To do Qualite : 0.27754501378681184 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Qualipapia_P20637017_18-02-2025_13_43_52.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20637017 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20637017 AND mptpi.`type`=4230 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20642413 order by id desc limit 1 Qualite : 0.23259470163861626 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20642414 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20642414 AND mptpi.`type`=4230 To do Qualite : 0.26574040135188726 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20642415 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20642415 AND mptpi.`type`=4230 To do Qualite : 0.2721513611920241 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20642416 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20642416 AND mptpi.`type`=4230 To do Qualite : 0.3007661507644513 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20642417 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20642417 AND mptpi.`type`=4230 To do Qualite : 0.24420278078145188 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20642418 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20642418 AND mptpi.`type`=4230 To do Qualite : 0.21776103208502326 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20642419 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20642419 AND mptpi.`type`=4230 To do Qualite : 0.2881777565033804 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20642420 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20642420 AND mptpi.`type`=4230 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20643604 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20643605 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20643606 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20643607 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20643608 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20643609 order by id desc limit 1 Qualite : 0.25319821387859964 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20643610 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20643610 AND mptpi.`type`=4230 To do Qualite : 0.25894522239794787 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20643611 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20643611 AND mptpi.`type`=4230 To do Qualite : 0.2508998009873564 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20643612 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20643612 AND mptpi.`type`=4230 To do Qualite : 0.23892473097152184 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20643613 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20643613 AND mptpi.`type`=4230 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20646049 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20646050 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20646051 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20646052 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20646053 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20646054 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20646055 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20646056 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20646057 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20646058 order by id desc limit 1 Qualite : 0.24120112796986734 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20646059 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20646059 AND mptpi.`type`=4230 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20646060 order by id desc limit 1 Qualite : 0.24965966001459144 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20646061 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20646061 AND mptpi.`type`=4230 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20647042 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20647043 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20647044 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20647045 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20647046 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20647047 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20647048 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20647049 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20647050 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20647051 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20647052 order by id desc limit 1 Qualite : 0.22425907292118422 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20647053 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20647053 AND mptpi.`type`=4230 To do Qualite : 0.21570964718041385 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20647054 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20647054 AND mptpi.`type`=4230 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20648162 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20648163 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20648164 order by id desc limit 1 Qualite : 0.27055926134380387 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20648165 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20648165 AND mptpi.`type`=4230 To do Qualite : 0.24943321893156153 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20648166 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20648166 AND mptpi.`type`=4230 To do Qualite : 0.2767826739313674 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20648167 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20648167 AND mptpi.`type`=4230 To do Qualite : 0.2992580761838743 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20648168 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20648168 AND mptpi.`type`=4230 To do Qualite : 0.2927493459833559 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20648169 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20648169 AND mptpi.`type`=4230 To do Qualite : 0.28379025747307535 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20648170 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11415 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11419 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11419 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11416 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11417 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11417 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11422 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 2 of step 11418 have datatype=6 We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11419 doesn't seem to be define in the database( WARNING : output 1 of step 11415 have datatype=7 whereas input 2 of step 11419 have datatype=None WARNING : type of output 3 of step 11419 doesn't seem to be define in the database( WARNING : type of input 1 of step 11416 doesn't seem to be define in the database( WARNING : type of output 1 of step 11416 doesn't seem to be define in the database( WARNING : type of input 3 of step 11417 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11416 have datatype=10 whereas input 0 of step 11420 have datatype=18 WARNING : type of input 5 of step 11418 doesn't seem to be define in the database( WARNING : output 0 of step 11420 have datatype=11 whereas input 5 of step 11418 have datatype=None WARNING : type of input 2 of step 11416 doesn't seem to be define in the database( WARNING : output 0 of step 11421 have datatype=5 whereas input 2 of step 11416 have datatype=None WARNING : output 0 of step 11418 have datatype=10 whereas input 0 of step 11422 have datatype=18 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20648170 AND mptpi.`type`=4230 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'18022025': {'nb_upload': 2, 'nb_taggue_class': 2, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1338370175, 1338370174] Looping around the photos to save general results len do output : 1 /20648165Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4234', None, None, None, None, None, None, None, '2597333') ('4234', '20648165', '1338370175', None, None, None, None, None, '2597333') ('4234', None, None, None, None, None, None, None, '2597333') ('4234', '20648165', '1338370174', None, None, None, None, None, '2597333') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 time used for this insertion : 0.012477874755859375 save_final save missing photos in datou_result : time spend for datou_step_exec : 41.77074980735779 time spend to save output : 0.012624263763427734 total time spend for step 11 : 41.783374071121216 caffe_path_current : About to save ! 2 After save, about to update current ! ret : 1 len(input) + len(total_photo_id_missing) : 2 set_done_treatment 32.76user 24.68system 2:58.68elapsed 32%CPU (0avgtext+0avgdata 2455348maxresident)k 705576inputs+10064outputs (268major+2090965minor)pagefaults 0swaps