python /home/admin/mtr/script_for_cron.py -j default -m 20 -a 'python3 ~/workarea/git/Velours/python/prod/datou.py -j batch_current -C 2501423' -s traitement_3459 -M 0 -S 0 -U 100,80,95 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/home/admin/workarea/git/apy', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3080549 load datou : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : step 0 init_dummy_multi_datou is not linked in the step_by_step architecture ! WARNING : step 1294 init_dummy_multi_datou is not linked in the step_by_step architecture ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : (photo_id, hashtag_id, score_max) was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2501423'] with mtr_portfolio_ids : ['19804580'] and first list_photo_ids : [] new path : /proc/3080549/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score over limit max, limiting to limit_max 20 list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 20 ; length of list_pids : 20 ; length of list_args : 20 time to download the photos : 3.2389092445373535 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Wed Feb 5 09:02:58 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10774 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-05 09:03:01.186953: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-05 09:03:01.211138: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-05 09:03:01.213265: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f9980000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-05 09:03:01.213326: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-05 09:03:01.217347: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-05 09:03:01.462035: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x356cf8f0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-05 09:03:01.462117: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-05 09:03:01.463666: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-05 09:03:01.464568: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-05 09:03:01.470447: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-05 09:03:01.473846: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-05 09:03:01.475241: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-05 09:03:01.479220: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-05 09:03:01.480845: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-05 09:03:01.487539: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-05 09:03:01.489180: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-05 09:03:01.489289: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-05 09:03:01.490109: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-05 09:03:01.490127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-05 09:03:01.490158: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-05 09:03:01.491613: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9985 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-05 09:03:01.801175: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-05 09:03:01.801290: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-05 09:03:01.801310: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-05 09:03:01.801329: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-05 09:03:01.801347: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-05 09:03:01.801365: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-05 09:03:01.801382: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-05 09:03:01.801400: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-05 09:03:01.802960: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-05 09:03:01.804300: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-05 09:03:01.804336: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-05 09:03:01.804355: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-05 09:03:01.804371: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-05 09:03:01.804388: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-05 09:03:01.804405: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-05 09:03:01.804421: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-05 09:03:01.804438: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-05 09:03:01.805971: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-05 09:03:01.806009: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-05 09:03:01.806020: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-05 09:03:01.806029: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-05 09:03:01.807642: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9985 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-05 09:03:08.764720: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-05 09:03:08.954866: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 20 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 6 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 2 Detection mask done ! Trying to reset tf kernel 3080702 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5706 tf kernel not reseted sub process len(results) : 20 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 20 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 10998 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.0005292892456054688 nb_pixel_total : 29719 time to create 1 rle with old method : 0.03522324562072754 length of segment : 179 time for calcul the mask position with numpy : 7.2479248046875e-05 nb_pixel_total : 1572 time to create 1 rle with old method : 0.0021178722381591797 length of segment : 34 time for calcul the mask position with numpy : 0.01583385467529297 nb_pixel_total : 852993 time to create 1 rle with new method : 0.03703665733337402 length of segment : 1911 time for calcul the mask position with numpy : 0.00010895729064941406 nb_pixel_total : 2696 time to create 1 rle with old method : 0.003564119338989258 length of segment : 51 time for calcul the mask position with numpy : 0.005065441131591797 nb_pixel_total : 375854 time to create 1 rle with new method : 0.014522314071655273 length of segment : 1111 time for calcul the mask position with numpy : 0.00036072731018066406 nb_pixel_total : 23894 time to create 1 rle with old method : 0.027789592742919922 length of segment : 130 time for calcul the mask position with numpy : 0.0004551410675048828 nb_pixel_total : 32370 time to create 1 rle with old method : 0.03574943542480469 length of segment : 186 time for calcul the mask position with numpy : 0.0019032955169677734 nb_pixel_total : 72777 time to create 1 rle with old method : 0.0800635814666748 length of segment : 357 time for calcul the mask position with numpy : 0.007889986038208008 nb_pixel_total : 494945 time to create 1 rle with new method : 0.02254796028137207 length of segment : 1105 time for calcul the mask position with numpy : 0.017864465713500977 nb_pixel_total : 1253832 time to create 1 rle with new method : 0.03456616401672363 length of segment : 1790 time for calcul the mask position with numpy : 0.0012369155883789062 nb_pixel_total : 70120 time to create 1 rle with old method : 0.07904338836669922 length of segment : 419 time for calcul the mask position with numpy : 0.0060083866119384766 nb_pixel_total : 313305 time to create 1 rle with new method : 0.00914621353149414 length of segment : 1127 time for calcul the mask position with numpy : 0.006837129592895508 nb_pixel_total : 317354 time to create 1 rle with new method : 0.009386777877807617 length of segment : 1051 time for calcul the mask position with numpy : 0.01293802261352539 nb_pixel_total : 729075 time to create 1 rle with new method : 0.0376737117767334 length of segment : 1892 time for calcul the mask position with numpy : 0.0011005401611328125 nb_pixel_total : 80397 time to create 1 rle with old method : 0.09540915489196777 length of segment : 374 time for calcul the mask position with numpy : 0.0016078948974609375 nb_pixel_total : 123286 time to create 1 rle with old method : 0.14377260208129883 length of segment : 348 time for calcul the mask position with numpy : 0.004732608795166016 nb_pixel_total : 395742 time to create 1 rle with new method : 0.014645099639892578 length of segment : 1532 time for calcul the mask position with numpy : 0.01970195770263672 nb_pixel_total : 1107302 time to create 1 rle with new method : 0.03644132614135742 length of segment : 1152 time for calcul the mask position with numpy : 0.0004916191101074219 nb_pixel_total : 23775 time to create 1 rle with old method : 0.02855682373046875 length of segment : 262 time for calcul the mask position with numpy : 0.0006992816925048828 nb_pixel_total : 51579 time to create 1 rle with old method : 0.06940674781799316 length of segment : 330 time for calcul the mask position with numpy : 0.0001723766326904297 nb_pixel_total : 1622 time to create 1 rle with old method : 0.0021657943725585938 length of segment : 32 time for calcul the mask position with numpy : 0.0020599365234375 nb_pixel_total : 116899 time to create 1 rle with old method : 0.12377023696899414 length of segment : 322 time for calcul the mask position with numpy : 0.005074024200439453 nb_pixel_total : 319105 time to create 1 rle with new method : 0.0121917724609375 length of segment : 1028 time for calcul the mask position with numpy : 0.000751495361328125 nb_pixel_total : 11963 time to create 1 rle with old method : 0.0173337459564209 length of segment : 101 time for calcul the mask position with numpy : 0.018940210342407227 nb_pixel_total : 1421290 time to create 1 rle with new method : 0.03746819496154785 length of segment : 1600 time for calcul the mask position with numpy : 0.005102872848510742 nb_pixel_total : 170978 time to create 1 rle with new method : 0.00815725326538086 length of segment : 663 time for calcul the mask position with numpy : 0.0001201629638671875 nb_pixel_total : 813 time to create 1 rle with old method : 0.0010209083557128906 length of segment : 56 time for calcul the mask position with numpy : 0.01772928237915039 nb_pixel_total : 1181198 time to create 1 rle with new method : 0.04074692726135254 length of segment : 2064 time for calcul the mask position with numpy : 0.00016641616821289062 nb_pixel_total : 6975 time to create 1 rle with old method : 0.00860452651977539 length of segment : 68 time for calcul the mask position with numpy : 0.00033664703369140625 nb_pixel_total : 19533 time to create 1 rle with old method : 0.0226590633392334 length of segment : 254 time for calcul the mask position with numpy : 0.0005695819854736328 nb_pixel_total : 40363 time to create 1 rle with old method : 0.04562830924987793 length of segment : 284 time for calcul the mask position with numpy : 0.0025191307067871094 nb_pixel_total : 226894 time to create 1 rle with new method : 0.004911184310913086 length of segment : 562 time for calcul the mask position with numpy : 0.00046706199645996094 nb_pixel_total : 12745 time to create 1 rle with old method : 0.014648675918579102 length of segment : 175 time for calcul the mask position with numpy : 0.015684843063354492 nb_pixel_total : 513481 time to create 1 rle with new method : 0.029199838638305664 length of segment : 1317 time for calcul the mask position with numpy : 7.843971252441406e-05 nb_pixel_total : 1933 time to create 1 rle with old method : 0.0025527477264404297 length of segment : 39 time for calcul the mask position with numpy : 0.00010895729064941406 nb_pixel_total : 4327 time to create 1 rle with old method : 0.005579471588134766 length of segment : 78 time for calcul the mask position with numpy : 0.0005254745483398438 nb_pixel_total : 29707 time to create 1 rle with old method : 0.034537553787231445 length of segment : 188 time for calcul the mask position with numpy : 0.0031914710998535156 nb_pixel_total : 127039 time to create 1 rle with old method : 0.14853811264038086 length of segment : 558 time for calcul the mask position with numpy : 0.0011005401611328125 nb_pixel_total : 60418 time to create 1 rle with old method : 0.06854557991027832 length of segment : 532 time for calcul the mask position with numpy : 0.0010595321655273438 nb_pixel_total : 70928 time to create 1 rle with old method : 0.08060479164123535 length of segment : 196 time for calcul the mask position with numpy : 0.010339975357055664 nb_pixel_total : 610168 time to create 1 rle with new method : 0.02718210220336914 length of segment : 1173 time for calcul the mask position with numpy : 0.000270843505859375 nb_pixel_total : 15808 time to create 1 rle with old method : 0.018093347549438477 length of segment : 332 time for calcul the mask position with numpy : 5.14984130859375e-05 nb_pixel_total : 1044 time to create 1 rle with old method : 0.0013661384582519531 length of segment : 48 time for calcul the mask position with numpy : 0.02275252342224121 nb_pixel_total : 1446259 time to create 1 rle with new method : 0.03580331802368164 length of segment : 1954 time for calcul the mask position with numpy : 0.00012612342834472656 nb_pixel_total : 5465 time to create 1 rle with old method : 0.006537914276123047 length of segment : 124 time for calcul the mask position with numpy : 0.0028243064880371094 nb_pixel_total : 268635 time to create 1 rle with new method : 0.0063076019287109375 length of segment : 1427 time for calcul the mask position with numpy : 0.00011682510375976562 nb_pixel_total : 4472 time to create 1 rle with old method : 0.00536656379699707 length of segment : 126 time for calcul the mask position with numpy : 0.0038928985595703125 nb_pixel_total : 328864 time to create 1 rle with new method : 0.009792566299438477 length of segment : 1313 time for calcul the mask position with numpy : 0.000408172607421875 nb_pixel_total : 6672 time to create 1 rle with old method : 0.007976531982421875 length of segment : 241 time for calcul the mask position with numpy : 0.009993791580200195 nb_pixel_total : 482839 time to create 1 rle with new method : 0.02694106101989746 length of segment : 1238 time for calcul the mask position with numpy : 0.0006494522094726562 nb_pixel_total : 34249 time to create 1 rle with old method : 0.046999216079711914 length of segment : 138 time for calcul the mask position with numpy : 0.004370212554931641 nb_pixel_total : 334867 time to create 1 rle with new method : 0.01677560806274414 length of segment : 1060 time for calcul the mask position with numpy : 0.0002346038818359375 nb_pixel_total : 1635 time to create 1 rle with old method : 0.003080606460571289 length of segment : 70 time for calcul the mask position with numpy : 0.03142499923706055 nb_pixel_total : 1162131 time to create 1 rle with new method : 0.04626750946044922 length of segment : 1988 time for calcul the mask position with numpy : 0.0036656856536865234 nb_pixel_total : 295095 time to create 1 rle with new method : 0.008181095123291016 length of segment : 1296 time for calcul the mask position with numpy : 0.012963294982910156 nb_pixel_total : 784499 time to create 1 rle with new method : 0.03407883644104004 length of segment : 1560 time for calcul the mask position with numpy : 0.0015869140625 nb_pixel_total : 78968 time to create 1 rle with old method : 0.1114344596862793 length of segment : 533 time for calcul the mask position with numpy : 0.0055196285247802734 nb_pixel_total : 332503 time to create 1 rle with new method : 0.011694908142089844 length of segment : 1172 time for calcul the mask position with numpy : 0.012449502944946289 nb_pixel_total : 940686 time to create 1 rle with new method : 0.03203082084655762 length of segment : 1614 time for calcul the mask position with numpy : 0.0005934238433837891 nb_pixel_total : 37320 time to create 1 rle with old method : 0.043951988220214844 length of segment : 207 time for calcul the mask position with numpy : 0.00403904914855957 nb_pixel_total : 312539 time to create 1 rle with new method : 0.010539531707763672 length of segment : 1091 time for calcul the mask position with numpy : 0.005189418792724609 nb_pixel_total : 336958 time to create 1 rle with new method : 0.010152339935302734 length of segment : 1009 time for calcul the mask position with numpy : 0.013486623764038086 nb_pixel_total : 1002665 time to create 1 rle with new method : 0.03390312194824219 length of segment : 1981 time for calcul the mask position with numpy : 0.00792384147644043 nb_pixel_total : 385636 time to create 1 rle with new method : 0.014448404312133789 length of segment : 1827 time for calcul the mask position with numpy : 0.01656174659729004 nb_pixel_total : 960032 time to create 1 rle with new method : 0.03392171859741211 length of segment : 2020 time for calcul the mask position with numpy : 0.0001533031463623047 nb_pixel_total : 7302 time to create 1 rle with old method : 0.008650064468383789 length of segment : 95 time for calcul the mask position with numpy : 0.0047299861907958984 nb_pixel_total : 418466 time to create 1 rle with new method : 0.012402772903442383 length of segment : 1887 time for calcul the mask position with numpy : 0.01139211654663086 nb_pixel_total : 586168 time to create 1 rle with new method : 0.025004863739013672 length of segment : 1981 time for calcul the mask position with numpy : 0.0014808177947998047 nb_pixel_total : 88097 time to create 1 rle with old method : 0.09905767440795898 length of segment : 468 time for calcul the mask position with numpy : 0.015197277069091797 nb_pixel_total : 1000360 time to create 1 rle with new method : 0.038533926010131836 length of segment : 1964 time for calcul the mask position with numpy : 0.003000974655151367 nb_pixel_total : 104544 time to create 1 rle with old method : 0.1180875301361084 length of segment : 495 time for calcul the mask position with numpy : 0.0008389949798583984 nb_pixel_total : 48176 time to create 1 rle with old method : 0.05661964416503906 length of segment : 409 time for calcul the mask position with numpy : 0.008098840713500977 nb_pixel_total : 374335 time to create 1 rle with new method : 0.017210006713867188 length of segment : 986 time for calcul the mask position with numpy : 0.001203775405883789 nb_pixel_total : 89901 time to create 1 rle with old method : 0.11991095542907715 length of segment : 164 time for calcul the mask position with numpy : 0.0007281303405761719 nb_pixel_total : 37274 time to create 1 rle with old method : 0.041439056396484375 length of segment : 318 time for calcul the mask position with numpy : 0.01527714729309082 nb_pixel_total : 1207819 time to create 1 rle with new method : 0.03491473197937012 length of segment : 1807 time spent for convertir_results : 12.629549503326416 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 76 chid ids of type : 3663 Number RLEs to save : 61574 save missing photos in datou_result : time spend for datou_step_exec : 33.2683846950531 time spend to save output : 3.373650550842285 total time spend for step 1 : 36.642035245895386 step2:crop_condition Wed Feb 5 09:03:35 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 20 ! batch 1 Loaded 76 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 10 About to insert : list_path_to_insert length 10 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1330517409, 1330517407, 1330517405, 1330517303, 1330517301, 1330517228, 1330517224, 1330517214, 1330517210, 1330516960, 1330516957, 1330516953, 1330516937, 1330516688, 1330516685, 1330516682, 1330516680, 1330514948, 1330514942, 1330514941] Looping around the photos to save general results len do output : 19 /-3655549013Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655549016Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655549014Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655549020Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655549047Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655549045Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655549078Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655549041Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655549027Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655549028Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655549031Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655549043Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655549049Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655549048Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655549063Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655549072Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655549086Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655549087Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655549042Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517409', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517407', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517405', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517303', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517301', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517228', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517224', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517214', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517210', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516960', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516957', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516953', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516937', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516688', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516685', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516682', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516680', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330514948', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330514942', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330514941', None, None, None, None, None, '2501423') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 77 time used for this insertion : 0.027958393096923828 save_final save missing photos in datou_result : time spend for datou_step_exec : 5.2749035358428955 time spend to save output : 0.028864383697509766 total time spend for step 2 : 5.303767919540405 step3:thcl Wed Feb 5 09:03:40 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.005492687225341797 time to convert the images to numpy array : 0.006694793701171875 time to import caffe and check if the image exist : 0.006627559661865234 time to convert the images to numpy array : 0.0051615238189697266 time to import caffe and check if the image exist : 0.005196332931518555 time to convert the images to numpy array : 0.007786750793457031 time to import caffe and check if the image exist : 0.005265235900878906 time to convert the images to numpy array : 0.00854349136352539 time to import caffe and check if the image exist : 0.005116701126098633 time to convert the images to numpy array : 0.010373353958129883 time to import caffe and check if the image exist : 0.008620500564575195 time to convert the images to numpy array : 0.012331247329711914 time to import caffe and check if the image exist : 0.006951808929443359 time to convert the images to numpy array : 0.01468968391418457 time to import caffe and check if the image exist : 0.008301019668579102 time to convert the images to numpy array : 0.015996217727661133 time to import caffe and check if the image exist : 0.006670951843261719 time to convert the images to numpy array : 0.01920008659362793 time to import caffe and check if the image exist : 0.008291959762573242 time to convert the images to numpy array : 0.02281951904296875 total time to convert the images to numpy array : 0.23886799812316895 list photo_ids error: [] list photo_ids correct : [-3655549047, -3655549045, -3655549042, -3655549049, -3655549048, -3655549078, -3655549041, -3655549013, -3655549016, -3655549063, -3655549072, -3655549031, -3655549043, -3655549086, -3655549087, -3655549014, -3655549020, -3655549027, -3655549028] number of photos to traite : 19 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10998 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/:/home/admin/workarea/git/apy/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10776 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.16292023658752441 time used to do the prediction : 0.07953882217407227 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10774 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/:/home/admin/workarea/git/apy/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10774 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.1425638198852539 time used to do the prediction : 0.05623507499694824 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 9.775161743164062e-06 save missing photos in datou_result : time spend for datou_step_exec : 7.609539747238159 time spend to save output : 0.0004317760467529297 total time spend for step 3 : 7.609971523284912 step4:merge_mask_thcl_custom Wed Feb 5 09:03:48 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 76 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1330517409, 1330517407, 1330517405, 1330517303, 1330517301, 1330517228, 1330517224, 1330517214, 1330517210, 1330516960, 1330516957, 1330516953, 1330516937, 1330516688, 1330516685, 1330516682, 1330516680, 1330514948, 1330514942, 1330514941] Looping around the photos to save general results len do output : 20 /1330517409Didn't retrieve data .Didn't retrieve data . /1330517407Didn't retrieve data .Didn't retrieve data . /1330517405Didn't retrieve data .Didn't retrieve data . /1330517303Didn't retrieve data .Didn't retrieve data . /1330517301Didn't retrieve data .Didn't retrieve data . /1330517228Didn't retrieve data .Didn't retrieve data . /1330517224Didn't retrieve data .Didn't retrieve data . /1330517214Didn't retrieve data .Didn't retrieve data . /1330517210Didn't retrieve data .Didn't retrieve data . /1330516960Didn't retrieve data .Didn't retrieve data . /1330516957Didn't retrieve data .Didn't retrieve data . /1330516953Didn't retrieve data .Didn't retrieve data . /1330516937Didn't retrieve data .Didn't retrieve data . /1330516688Didn't retrieve data .Didn't retrieve data . /1330516685Didn't retrieve data .Didn't retrieve data . /1330516682Didn't retrieve data .Didn't retrieve data . /1330516680Didn't retrieve data .Didn't retrieve data . /1330514948Didn't retrieve data .Didn't retrieve data . /1330514942Didn't retrieve data .Didn't retrieve data . /1330514941Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517409', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517407', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517405', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517303', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517301', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517228', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517224', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517214', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517210', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516960', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516957', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516953', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516937', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516688', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516685', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516682', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516680', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330514948', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330514942', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330514941', None, None, None, None, None, '2501423') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 60 time used for this insertion : 0.01582932472229004 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.07407832145690918 time spend to save output : 0.016645193099975586 total time spend for step 4 : 0.09072351455688477 step5:rle_unique_nms_with_priority Wed Feb 5 09:03:48 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 5 nb_hashtags : 3 time to prepare the origin masks : 0.41550779342651367 time for calcul the mask position with numpy : 0.014870643615722656 nb_pixel_total : 1005049 time to create 1 rle with new method : 0.05146288871765137 time for calcul the mask position with numpy : 0.006273984909057617 nb_pixel_total : 1561 time to create 1 rle with old method : 0.0019462108612060547 time for calcul the mask position with numpy : 0.0067195892333984375 nb_pixel_total : 134587 time to create 1 rle with old method : 0.15538692474365234 time for calcul the mask position with numpy : 0.013285636901855469 nb_pixel_total : 900135 time to create 1 rle with new method : 0.04069185256958008 time for calcul the mask position with numpy : 0.0059583187103271484 nb_pixel_total : 2685 time to create 1 rle with old method : 0.0030770301818847656 time for calcul the mask position with numpy : 0.00635838508605957 nb_pixel_total : 29583 time to create 1 rle with old method : 0.032740116119384766 create new chi : 0.3475687503814697 time to delete rle : 0.023702383041381836 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 6169 TO DO : save crop sub photo not yet done ! save time : 0.3706543445587158 nb_obj : 4 nb_hashtags : 1 time to prepare the origin masks : 0.35642409324645996 time for calcul the mask position with numpy : 0.019326210021972656 nb_pixel_total : 1435382 time to create 1 rle with new method : 0.04339098930358887 time for calcul the mask position with numpy : 0.011937379837036133 nb_pixel_total : 509725 time to create 1 rle with new method : 0.03286409378051758 time for calcul the mask position with numpy : 0.006684064865112305 nb_pixel_total : 32314 time to create 1 rle with old method : 0.03680729866027832 time for calcul the mask position with numpy : 0.00725865364074707 nb_pixel_total : 72464 time to create 1 rle with old method : 0.08023929595947266 time for calcul the mask position with numpy : 0.006178617477416992 nb_pixel_total : 23715 time to create 1 rle with old method : 0.027878522872924805 create new chi : 0.277026891708374 time to delete rle : 0.0004086494445800781 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 4374 TO DO : save crop sub photo not yet done ! save time : 0.40049290657043457 nb_obj : 3 nb_hashtags : 1 time to prepare the origin masks : 0.398845911026001 time for calcul the mask position with numpy : 0.011008024215698242 nb_pixel_total : 712151 time to create 1 rle with new method : 0.03776669502258301 time for calcul the mask position with numpy : 0.0063593387603759766 nb_pixel_total : 66595 time to create 1 rle with old method : 0.07432055473327637 time for calcul the mask position with numpy : 0.006073474884033203 nb_pixel_total : 19283 time to create 1 rle with old method : 0.021950483322143555 time for calcul the mask position with numpy : 0.01805281639099121 nb_pixel_total : 1275571 time to create 1 rle with new method : 0.0375823974609375 create new chi : 0.21878385543823242 time to delete rle : 0.000457763671875 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 5449 TO DO : save crop sub photo not yet done ! save time : 0.31873512268066406 nb_obj : 5 nb_hashtags : 3 time to prepare the origin masks : 0.7027015686035156 time for calcul the mask position with numpy : 0.012367963790893555 nb_pixel_total : 818860 time to create 1 rle with new method : 0.04575777053833008 time for calcul the mask position with numpy : 0.00795435905456543 nb_pixel_total : 174540 time to create 1 rle with new method : 0.03818178176879883 time for calcul the mask position with numpy : 0.0065784454345703125 nb_pixel_total : 50731 time to create 1 rle with old method : 0.06144380569458008 time for calcul the mask position with numpy : 0.012758731842041016 nb_pixel_total : 631862 time to create 1 rle with new method : 0.042966604232788086 time for calcul the mask position with numpy : 0.008739709854125977 nb_pixel_total : 317325 time to create 1 rle with new method : 0.03706765174865723 time for calcul the mask position with numpy : 0.006336212158203125 nb_pixel_total : 80282 time to create 1 rle with old method : 0.08717632293701172 create new chi : 0.3797786235809326 time to delete rle : 0.0007681846618652344 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 8665 TO DO : save crop sub photo not yet done ! save time : 0.5735287666320801 nb_obj : 5 nb_hashtags : 2 time to prepare the origin masks : 0.5588586330413818 time for calcul the mask position with numpy : 0.0160062313079834 nb_pixel_total : 811345 time to create 1 rle with new method : 0.05451846122741699 time for calcul the mask position with numpy : 0.006958961486816406 nb_pixel_total : 23490 time to create 1 rle with old method : 0.029685258865356445 time for calcul the mask position with numpy : 0.007168769836425781 nb_pixel_total : 48983 time to create 1 rle with old method : 0.056360483169555664 time for calcul the mask position with numpy : 0.007569074630737305 nb_pixel_total : 25361 time to create 1 rle with old method : 0.028625965118408203 time for calcul the mask position with numpy : 0.0062100887298583984 nb_pixel_total : 51485 time to create 1 rle with old method : 0.05657029151916504 time for calcul the mask position with numpy : 0.015276432037353516 nb_pixel_total : 1112936 time to create 1 rle with new method : 0.04284214973449707 create new chi : 0.3374333381652832 time to delete rle : 0.0005164146423339844 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 5349 TO DO : save crop sub photo not yet done ! save time : 0.30989837646484375 nb_obj : 4 nb_hashtags : 1 time to prepare the origin masks : 0.3551139831542969 time for calcul the mask position with numpy : 0.010217666625976562 nb_pixel_total : 576631 time to create 1 rle with new method : 0.03224372863769531 time for calcul the mask position with numpy : 0.00589752197265625 nb_pixel_total : 800 time to create 1 rle with old method : 0.0009613037109375 time for calcul the mask position with numpy : 0.0059871673583984375 nb_pixel_total : 47878 time to create 1 rle with old method : 0.05310821533203125 time for calcul the mask position with numpy : 0.015006780624389648 nb_pixel_total : 1436388 time to create 1 rle with new method : 0.027840852737426758 time for calcul the mask position with numpy : 0.0056629180908203125 nb_pixel_total : 11903 time to create 1 rle with old method : 0.012305736541748047 create new chi : 0.170670747756958 time to delete rle : 0.0004200935363769531 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 4691 TO DO : save crop sub photo not yet done ! save time : 0.2818896770477295 nb_obj : 5 nb_hashtags : 4 time to prepare the origin masks : 0.7036521434783936 time for calcul the mask position with numpy : 0.012715339660644531 nb_pixel_total : 723503 time to create 1 rle with new method : 0.03934144973754883 time for calcul the mask position with numpy : 0.0062961578369140625 nb_pixel_total : 19146 time to create 1 rle with old method : 0.021202564239501953 time for calcul the mask position with numpy : 0.006845712661743164 nb_pixel_total : 67019 time to create 1 rle with old method : 0.0805673599243164 time for calcul the mask position with numpy : 0.006750583648681641 nb_pixel_total : 6796 time to create 1 rle with old method : 0.010890722274780273 time for calcul the mask position with numpy : 0.007058858871459961 nb_pixel_total : 39096 time to create 1 rle with old method : 0.05341172218322754 time for calcul the mask position with numpy : 0.01865100860595703 nb_pixel_total : 1218040 time to create 1 rle with new method : 0.03294658660888672 create new chi : 0.3008887767791748 time to delete rle : 0.0005831718444824219 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 6049 TO DO : save crop sub photo not yet done ! save time : 0.6085879802703857 nb_obj : 7 nb_hashtags : 5 time to prepare the origin masks : 0.5863900184631348 time for calcul the mask position with numpy : 0.020462751388549805 nb_pixel_total : 1406951 time to create 1 rle with new method : 0.044057607650756836 time for calcul the mask position with numpy : 0.0066416263580322266 nb_pixel_total : 29333 time to create 1 rle with old method : 0.03493237495422363 time for calcul the mask position with numpy : 0.00622105598449707 nb_pixel_total : 1919 time to create 1 rle with old method : 0.003139495849609375 time for calcul the mask position with numpy : 0.006868600845336914 nb_pixel_total : 32233 time to create 1 rle with old method : 0.03592276573181152 time for calcul the mask position with numpy : 0.006397247314453125 nb_pixel_total : 72865 time to create 1 rle with old method : 0.07971644401550293 time for calcul the mask position with numpy : 0.006051063537597656 nb_pixel_total : 4288 time to create 1 rle with old method : 0.005091428756713867 time for calcul the mask position with numpy : 0.005860805511474609 nb_pixel_total : 12719 time to create 1 rle with old method : 0.014187812805175781 time for calcul the mask position with numpy : 0.008854866027832031 nb_pixel_total : 513292 time to create 1 rle with new method : 0.03368043899536133 create new chi : 0.32158613204956055 time to delete rle : 0.00061798095703125 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 6435 TO DO : save crop sub photo not yet done ! save time : 0.5266902446746826 nb_obj : 4 nb_hashtags : 1 time to prepare the origin masks : 0.36035680770874023 time for calcul the mask position with numpy : 0.022919178009033203 nb_pixel_total : 1361996 time to create 1 rle with new method : 0.04028439521789551 time for calcul the mask position with numpy : 0.00617671012878418 nb_pixel_total : 1034 time to create 1 rle with old method : 0.0013167858123779297 time for calcul the mask position with numpy : 0.00612187385559082 nb_pixel_total : 15781 time to create 1 rle with old method : 0.017109155654907227 time for calcul the mask position with numpy : 0.012991189956665039 nb_pixel_total : 623950 time to create 1 rle with new method : 0.027564525604248047 time for calcul the mask position with numpy : 0.006438255310058594 nb_pixel_total : 70839 time to create 1 rle with old method : 0.07883477210998535 create new chi : 0.221177339553833 time to delete rle : 0.0004405975341796875 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 4172 TO DO : save crop sub photo not yet done ! save time : 0.2760791778564453 nb_obj : 4 nb_hashtags : 1 time to prepare the origin masks : 0.5109753608703613 time for calcul the mask position with numpy : 0.007888078689575195 nb_pixel_total : 277227 time to create 1 rle with new method : 0.02994704246520996 time for calcul the mask position with numpy : 0.006242990493774414 nb_pixel_total : 63211 time to create 1 rle with old method : 0.06991338729858398 time for calcul the mask position with numpy : 0.006230831146240234 nb_pixel_total : 68082 time to create 1 rle with old method : 0.07638931274414062 time for calcul the mask position with numpy : 0.00606989860534668 nb_pixel_total : 5453 time to create 1 rle with old method : 0.006005764007568359 time for calcul the mask position with numpy : 0.016954421997070312 nb_pixel_total : 1659627 time to create 1 rle with new method : 0.0273282527923584 create new chi : 0.2537860870361328 time to delete rle : 0.0005424022674560547 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 5579 TO DO : save crop sub photo not yet done ! save time : 0.4011876583099365 nb_obj : 4 nb_hashtags : 2 time to prepare the origin masks : 0.4382774829864502 time for calcul the mask position with numpy : 0.01878976821899414 nb_pixel_total : 1401948 time to create 1 rle with new method : 0.04355192184448242 time for calcul the mask position with numpy : 0.00646209716796875 nb_pixel_total : 34123 time to create 1 rle with old method : 0.038514137268066406 time for calcul the mask position with numpy : 0.006836652755737305 nb_pixel_total : 130337 time to create 1 rle with old method : 0.16779518127441406 time for calcul the mask position with numpy : 0.014224529266357422 nb_pixel_total : 500548 time to create 1 rle with new method : 0.04995107650756836 time for calcul the mask position with numpy : 0.0077817440032958984 nb_pixel_total : 6644 time to create 1 rle with old method : 0.007598161697387695 create new chi : 0.3696467876434326 time to delete rle : 0.0006012916564941406 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 4953 TO DO : save crop sub photo not yet done ! save time : 0.2803332805633545 nb_obj : 3 nb_hashtags : 1 time to prepare the origin masks : 0.48079657554626465 time for calcul the mask position with numpy : 0.011224746704101562 nb_pixel_total : 718103 time to create 1 rle with new method : 0.03286242485046387 time for calcul the mask position with numpy : 0.006517648696899414 nb_pixel_total : 47438 time to create 1 rle with old method : 0.05173301696777344 time for calcul the mask position with numpy : 0.016880512237548828 nb_pixel_total : 1306428 time to create 1 rle with new method : 0.03770804405212402 time for calcul the mask position with numpy : 0.0063629150390625 nb_pixel_total : 1631 time to create 1 rle with old method : 0.0019621849060058594 create new chi : 0.16985559463500977 time to delete rle : 0.00040030479431152344 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 4189 TO DO : save crop sub photo not yet done ! save time : 0.39519786834716797 nb_obj : 2 nb_hashtags : 1 time to prepare the origin masks : 0.268479585647583 time for calcul the mask position with numpy : 0.019012451171875 nb_pixel_total : 1240836 time to create 1 rle with new method : 0.04244518280029297 time for calcul the mask position with numpy : 0.006169557571411133 nb_pixel_total : 27374 time to create 1 rle with old method : 0.030370712280273438 time for calcul the mask position with numpy : 0.012063741683959961 nb_pixel_total : 805390 time to create 1 rle with new method : 0.04023432731628418 create new chi : 0.1598970890045166 time to delete rle : 0.0005495548248291016 batch 1 Loaded 3 chid ids of type : 3726 Number RLEs to save : 4309 TO DO : save crop sub photo not yet done ! save time : 0.29509449005126953 nb_obj : 4 nb_hashtags : 2 time to prepare the origin masks : 0.42949509620666504 time for calcul the mask position with numpy : 0.01226496696472168 nb_pixel_total : 743107 time to create 1 rle with new method : 0.04372692108154297 time for calcul the mask position with numpy : 0.006213188171386719 nb_pixel_total : 37258 time to create 1 rle with old method : 0.04484081268310547 time for calcul the mask position with numpy : 0.006868839263916016 nb_pixel_total : 168819 time to create 1 rle with new method : 0.03439927101135254 time for calcul the mask position with numpy : 0.010606527328491211 nb_pixel_total : 725592 time to create 1 rle with new method : 0.034780025482177734 time for calcul the mask position with numpy : 0.008044004440307617 nb_pixel_total : 398824 time to create 1 rle with new method : 0.033855438232421875 create new chi : 0.2462010383605957 time to delete rle : 0.0006842613220214844 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 7051 TO DO : save crop sub photo not yet done ! save time : 0.7092785835266113 nb_obj : 3 nb_hashtags : 1 time to prepare the origin masks : 0.29296135902404785 time for calcul the mask position with numpy : 0.01181173324584961 nb_pixel_total : 789833 time to create 1 rle with new method : 0.033883094787597656 time for calcul the mask position with numpy : 0.007237672805786133 nb_pixel_total : 180025 time to create 1 rle with new method : 0.03150510787963867 time for calcul the mask position with numpy : 0.011651039123535156 nb_pixel_total : 767676 time to create 1 rle with new method : 0.02898097038269043 time for calcul the mask position with numpy : 0.0084075927734375 nb_pixel_total : 336066 time to create 1 rle with new method : 0.029224872589111328 create new chi : 0.16471457481384277 time to delete rle : 0.0006389617919921875 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 7023 TO DO : save crop sub photo not yet done ! save time : 0.40550923347473145 nb_obj : 3 nb_hashtags : 2 time to prepare the origin masks : 0.3403892517089844 time for calcul the mask position with numpy : 0.01573634147644043 nb_pixel_total : 970435 time to create 1 rle with new method : 0.04637336730957031 time for calcul the mask position with numpy : 0.006361722946166992 nb_pixel_total : 3903 time to create 1 rle with old method : 0.004439115524291992 time for calcul the mask position with numpy : 0.006720066070556641 nb_pixel_total : 142619 time to create 1 rle with old method : 0.15852665901184082 time for calcul the mask position with numpy : 0.013909101486206055 nb_pixel_total : 956643 time to create 1 rle with new method : 0.04200887680053711 create new chi : 0.3032348155975342 time to delete rle : 0.0006687641143798828 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 6685 TO DO : save crop sub photo not yet done ! save time : 0.5265963077545166 nb_obj : 2 nb_hashtags : 1 time to prepare the origin masks : 0.2826344966888428 time for calcul the mask position with numpy : 0.014770746231079102 nb_pixel_total : 1291113 time to create 1 rle with new method : 0.036612510681152344 time for calcul the mask position with numpy : 0.00623774528503418 nb_pixel_total : 53917 time to create 1 rle with old method : 0.05849909782409668 time for calcul the mask position with numpy : 0.010952472686767578 nb_pixel_total : 728570 time to create 1 rle with new method : 0.02754044532775879 create new chi : 0.1572246551513672 time to delete rle : 0.0004432201385498047 batch 1 Loaded 3 chid ids of type : 3726 Number RLEs to save : 4656 TO DO : save crop sub photo not yet done ! save time : 0.29291653633117676 nb_obj : 3 nb_hashtags : 1 time to prepare the origin masks : 0.2832822799682617 time for calcul the mask position with numpy : 0.0159757137298584 nb_pixel_total : 1013754 time to create 1 rle with new method : 0.04162478446960449 time for calcul the mask position with numpy : 0.006244182586669922 nb_pixel_total : 16307 time to create 1 rle with old method : 0.017670392990112305 time for calcul the mask position with numpy : 0.006323099136352539 nb_pixel_total : 50215 time to create 1 rle with old method : 0.05442309379577637 time for calcul the mask position with numpy : 0.01284170150756836 nb_pixel_total : 993324 time to create 1 rle with new method : 0.04056215286254883 create new chi : 0.20415997505187988 time to delete rle : 0.0005767345428466797 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 6003 TO DO : save crop sub photo not yet done ! save time : 0.4684932231903076 nb_obj : 3 nb_hashtags : 2 time to prepare the origin masks : 0.19411897659301758 time for calcul the mask position with numpy : 0.02010035514831543 nb_pixel_total : 1573305 time to create 1 rle with new method : 0.040648698806762695 time for calcul the mask position with numpy : 0.006265878677368164 nb_pixel_total : 89550 time to create 1 rle with old method : 0.0960240364074707 time for calcul the mask position with numpy : 0.00636601448059082 nb_pixel_total : 37152 time to create 1 rle with old method : 0.04166579246520996 time for calcul the mask position with numpy : 0.007879495620727539 nb_pixel_total : 373593 time to create 1 rle with new method : 0.03414440155029297 create new chi : 0.2594785690307617 time to delete rle : 0.0004153251647949219 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 4002 TO DO : save crop sub photo not yet done ! save time : 0.3538639545440674 nb_obj : 1 nb_hashtags : 1 time to prepare the origin masks : 0.21628618240356445 time for calcul the mask position with numpy : 0.01305079460144043 nb_pixel_total : 689326 time to create 1 rle with new method : 0.03419137001037598 time for calcul the mask position with numpy : 0.01476287841796875 nb_pixel_total : 1384274 time to create 1 rle with new method : 0.02768683433532715 create new chi : 0.09010744094848633 time to delete rle : 0.0003161430358886719 batch 1 Loaded 2 chid ids of type : 3726 Number RLEs to save : 3580 TO DO : save crop sub photo not yet done ! save time : 0.23523759841918945 map_output_result : {1330517409: (0.0, 'Should be the crop_list due to order', 0.0), 1330517407: (0.0, 'Should be the crop_list due to order', 0.0), 1330517405: (0.0, 'Should be the crop_list due to order', 0.0), 1330517303: (0.0, 'Should be the crop_list due to order', 0.0), 1330517301: (0.0, 'Should be the crop_list due to order', 0.0), 1330517228: (0.0, 'Should be the crop_list due to order', 0.0), 1330517224: (0.0, 'Should be the crop_list due to order', 0.0), 1330517214: (0.0, 'Should be the crop_list due to order', 0.0), 1330517210: (0.0, 'Should be the crop_list due to order', 0.0), 1330516960: (0.0, 'Should be the crop_list due to order', 0.0), 1330516957: (0.0, 'Should be the crop_list due to order', 0.0), 1330516953: (0.0, 'Should be the crop_list due to order', 0.0), 1330516937: (0.0, 'Should be the crop_list due to order', 0.0), 1330516688: (0.0, 'Should be the crop_list due to order', 0.0), 1330516685: (0.0, 'Should be the crop_list due to order', 0.0), 1330516682: (0.0, 'Should be the crop_list due to order', 0.0), 1330516680: (0.0, 'Should be the crop_list due to order', 0.0), 1330514948: (0.0, 'Should be the crop_list due to order', 0.0), 1330514942: (0.0, 'Should be the crop_list due to order', 0.0), 1330514941: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1330517409, 1330517407, 1330517405, 1330517303, 1330517301, 1330517228, 1330517224, 1330517214, 1330517210, 1330516960, 1330516957, 1330516953, 1330516937, 1330516688, 1330516685, 1330516682, 1330516680, 1330514948, 1330514942, 1330514941] Looping around the photos to save general results len do output : 20 /1330517409.Didn't retrieve data . /1330517407.Didn't retrieve data . /1330517405.Didn't retrieve data . /1330517303.Didn't retrieve data . /1330517301.Didn't retrieve data . /1330517228.Didn't retrieve data . /1330517224.Didn't retrieve data . /1330517214.Didn't retrieve data . /1330517210.Didn't retrieve data . /1330516960.Didn't retrieve data . /1330516957.Didn't retrieve data . /1330516953.Didn't retrieve data . /1330516937.Didn't retrieve data . /1330516688.Didn't retrieve data . /1330516685.Didn't retrieve data . /1330516682.Didn't retrieve data . /1330516680.Didn't retrieve data . /1330514948.Didn't retrieve data . /1330514942.Didn't retrieve data . /1330514941.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517409', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517407', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517405', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517303', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517301', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517228', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517224', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517214', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517210', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516960', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516957', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516953', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516937', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516688', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516685', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516682', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516680', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330514948', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330514942', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330514941', None, None, None, None, None, '2501423') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 60 time used for this insertion : 0.0144195556640625 save_final save missing photos in datou_result : time spend for datou_step_exec : 22.001734256744385 time spend to save output : 0.015023469924926758 total time spend for step 5 : 22.01675772666931 step6:crop_condition Wed Feb 5 09:04:10 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 20 ! batch 1 Loaded 94 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738742651_3080549 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.7131361961364746 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738742652_3080549 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.5080118179321289 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738742653_3080549 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6368353366851807 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738742654_3080549 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.5012733936309814 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1330517409, 1330517407, 1330517405, 1330517303, 1330517301, 1330517228, 1330517224, 1330517214, 1330517210, 1330516960, 1330516957, 1330516953, 1330516937, 1330516688, 1330516685, 1330516682, 1330516680, 1330514948, 1330514942, 1330514941] Looping around the photos to save general results len do output : 4 /1334854258Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334854280Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334854303Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334854320Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517409', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517407', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517405', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517303', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517301', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517228', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517224', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517214', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517210', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516960', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516957', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516953', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516937', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516688', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516685', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516682', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516680', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330514948', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330514942', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330514941', None, None, None, None, None, '2501423') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 32 time used for this insertion : 0.022730588912963867 save_final save missing photos in datou_result : time spend for datou_step_exec : 7.727052211761475 time spend to save output : 0.02308058738708496 total time spend for step 6 : 7.75013279914856 step7:ventilate_hashtags_in_portfolio Wed Feb 5 09:04:17 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 19804580 get user id for portfolio 19804580 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19804580 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('cartonnette','kraft','autre_refus','Carton_brun','papier','mal_croppe','plastique','Teint_Dans_La_Masse','Carton_gris','metal','flou','environnement')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19804580 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('cartonnette','kraft','autre_refus','Carton_brun','papier','mal_croppe','plastique','Teint_Dans_La_Masse','Carton_gris','metal','flou','environnement')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19804580 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('cartonnette','kraft','autre_refus','Carton_brun','papier','mal_croppe','plastique','Teint_Dans_La_Masse','Carton_gris','metal','flou','environnement')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20248526,20248527,20248528,20248529,20248530,20248531,20248532,20248533,20248534,20248535,20248536,20248537?tags=cartonnette,kraft,autre_refus,Carton_brun,papier,mal_croppe,plastique,Teint_Dans_La_Masse,Carton_gris,metal,flou,environnement Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1330517409, 1330517407, 1330517405, 1330517303, 1330517301, 1330517228, 1330517224, 1330517214, 1330517210, 1330516960, 1330516957, 1330516953, 1330516937, 1330516688, 1330516685, 1330516682, 1330516680, 1330514948, 1330514942, 1330514941] Looping around the photos to save general results len do output : 1 /19804580. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517409', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517407', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517405', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517303', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517301', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517228', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517224', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517214', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517210', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516960', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516957', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516953', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516937', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516688', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516685', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516682', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516680', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330514948', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330514942', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330514941', None, None, None, None, None, '2501423') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 21 time used for this insertion : 0.3488311767578125 save_final save missing photos in datou_result : time spend for datou_step_exec : 2.409763813018799 time spend to save output : 0.3491644859313965 total time spend for step 7 : 2.7589282989501953 step8:final Wed Feb 5 09:04:20 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1330517409: ('0.019060807214082234',), 1330517407: ('0.019060807214082234',), 1330517405: ('0.019060807214082234',), 1330517303: ('0.019060807214082234',), 1330517301: ('0.019060807214082234',), 1330517228: ('0.019060807214082234',), 1330517224: ('0.019060807214082234',), 1330517214: ('0.019060807214082234',), 1330517210: ('0.019060807214082234',), 1330516960: ('0.019060807214082234',), 1330516957: ('0.019060807214082234',), 1330516953: ('0.019060807214082234',), 1330516937: ('0.019060807214082234',), 1330516688: ('0.019060807214082234',), 1330516685: ('0.019060807214082234',), 1330516682: ('0.019060807214082234',), 1330516680: ('0.019060807214082234',), 1330514948: ('0.019060807214082234',), 1330514942: ('0.019060807214082234',), 1330514941: ('0.019060807214082234',)} new output for save of step final : {1330517409: ('0.019060807214082234',), 1330517407: ('0.019060807214082234',), 1330517405: ('0.019060807214082234',), 1330517303: ('0.019060807214082234',), 1330517301: ('0.019060807214082234',), 1330517228: ('0.019060807214082234',), 1330517224: ('0.019060807214082234',), 1330517214: ('0.019060807214082234',), 1330517210: ('0.019060807214082234',), 1330516960: ('0.019060807214082234',), 1330516957: ('0.019060807214082234',), 1330516953: ('0.019060807214082234',), 1330516937: ('0.019060807214082234',), 1330516688: ('0.019060807214082234',), 1330516685: ('0.019060807214082234',), 1330516682: ('0.019060807214082234',), 1330516680: ('0.019060807214082234',), 1330514948: ('0.019060807214082234',), 1330514942: ('0.019060807214082234',), 1330514941: ('0.019060807214082234',)} [1330517409, 1330517407, 1330517405, 1330517303, 1330517301, 1330517228, 1330517224, 1330517214, 1330517210, 1330516960, 1330516957, 1330516953, 1330516937, 1330516688, 1330516685, 1330516682, 1330516680, 1330514948, 1330514942, 1330514941] Looping around the photos to save general results len do output : 20 /1330517409.Didn't retrieve data . /1330517407.Didn't retrieve data . /1330517405.Didn't retrieve data . /1330517303.Didn't retrieve data . /1330517301.Didn't retrieve data . /1330517228.Didn't retrieve data . /1330517224.Didn't retrieve data . /1330517214.Didn't retrieve data . /1330517210.Didn't retrieve data . /1330516960.Didn't retrieve data . /1330516957.Didn't retrieve data . /1330516953.Didn't retrieve data . /1330516937.Didn't retrieve data . /1330516688.Didn't retrieve data . /1330516685.Didn't retrieve data . /1330516682.Didn't retrieve data . /1330516680.Didn't retrieve data . /1330514948.Didn't retrieve data . /1330514942.Didn't retrieve data . /1330514941.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517409', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517407', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517405', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517303', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517301', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517228', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517224', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517214', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517210', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516960', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516957', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516953', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516937', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516688', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516685', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516682', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516680', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330514948', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330514942', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330514941', None, None, None, None, None, '2501423') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 60 time used for this insertion : 0.014101505279541016 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.11679434776306152 time spend to save output : 0.01517176628112793 total time spend for step 8 : 0.13196611404418945 step9:velours_tree Wed Feb 5 09:04:20 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.0313112735748291 time spend to save output : 3.0279159545898438e-05 total time spend for step 9 : 0.031341552734375 step10:send_mail_cod Wed Feb 5 09:04:20 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin in order to get the selector url, please entre the license of selector results_Auto_P19804580_05-02-2025_09_04_20.pdf 20248526 change filename to text .imagette202485261738742660 20248527 imagette202485271738742660 20248528 change filename to text .imagette202485281738742660 20248531 imagette202485311738742660 20248532 imagette202485321738742660 20248535 imagette202485351738742660 20248536 imagette202485361738742660 20248529 imagette202485291738742660 20248533 change filename to text .imagette202485331738742660 20248534 change filename to text .imagette202485341738742661 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=19804580 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20248526,20248527,20248528,20248529,20248530,20248531,20248532,20248533,20248534,20248535,20248536,20248537?tags=cartonnette,kraft,autre_refus,Carton_brun,papier,mal_croppe,plastique,Teint_Dans_La_Masse,Carton_gris,metal,flou,environnement your option no_mail is active, we will not send the real mail to your client args[1330517409] : ('0.019060807214082234',) no score found for photo 1330517409 We are sending mail with results at report@fotonower.com args[1330517407] : ('0.019060807214082234',) no score found for photo 1330517407 We are sending mail with results at report@fotonower.com args[1330517405] : ('0.019060807214082234',) no score found for photo 1330517405 We are sending mail with results at report@fotonower.com args[1330517303] : ('0.019060807214082234',) no score found for photo 1330517303 We are sending mail with results at report@fotonower.com args[1330517301] : ('0.019060807214082234',) no score found for photo 1330517301 We are sending mail with results at report@fotonower.com args[1330517228] : ('0.019060807214082234',) no score found for photo 1330517228 We are sending mail with results at report@fotonower.com args[1330517224] : ('0.019060807214082234',) no score found for photo 1330517224 We are sending mail with results at report@fotonower.com args[1330517214] : ('0.019060807214082234',) no score found for photo 1330517214 We are sending mail with results at report@fotonower.com args[1330517210] : ('0.019060807214082234',) no score found for photo 1330517210 We are sending mail with results at report@fotonower.com args[1330516960] : ('0.019060807214082234',) no score found for photo 1330516960 We are sending mail with results at report@fotonower.com args[1330516957] : ('0.019060807214082234',) no score found for photo 1330516957 We are sending mail with results at report@fotonower.com args[1330516953] : ('0.019060807214082234',) no score found for photo 1330516953 We are sending mail with results at report@fotonower.com args[1330516937] : ('0.019060807214082234',) no score found for photo 1330516937 We are sending mail with results at report@fotonower.com args[1330516688] : ('0.019060807214082234',) no score found for photo 1330516688 We are sending mail with results at report@fotonower.com args[1330516685] : ('0.019060807214082234',) no score found for photo 1330516685 We are sending mail with results at report@fotonower.com args[1330516682] : ('0.019060807214082234',) no score found for photo 1330516682 We are sending mail with results at report@fotonower.com args[1330516680] : ('0.019060807214082234',) no score found for photo 1330516680 We are sending mail with results at report@fotonower.com args[1330514948] : ('0.019060807214082234',) no score found for photo 1330514948 We are sending mail with results at report@fotonower.com args[1330514942] : ('0.019060807214082234',) no score found for photo 1330514942 We are sending mail with results at report@fotonower.com args[1330514941] : ('0.019060807214082234',) no score found for photo 1330514941 We are sending mail with results at report@fotonower.com refus_total : 0.019060807214082234 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=19804580 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1330514941,1330516688,1330517409,1330517407,1330517405,1330517303,1330517301,1330517228,1330517224,1330517214,1330517210,1330516960,1330516957,1330516953,1330516937,1330514555,1330516685,1330516682,1330516680,1330514948) Found this number of photos: 20 begin to download photo : 1330514941 begin to download photo : 1330517303 begin to download photo : 1330517210 begin to download photo : 1330514555 download finish for photo 1330514941 begin to download photo : 1330516688 download finish for photo 1330517303 begin to download photo : 1330517301 download finish for photo 1330514555 begin to download photo : 1330516685 download finish for photo 1330517210 begin to download photo : 1330516960 download finish for photo 1330516688 begin to download photo : 1330517409 download finish for photo 1330517301 begin to download photo : 1330517228 download finish for photo 1330516960 begin to download photo : 1330516957 download finish for photo 1330516685 begin to download photo : 1330516682 download finish for photo 1330517409 begin to download photo : 1330517407 download finish for photo 1330516682 begin to download photo : 1330516680 download finish for photo 1330517228 begin to download photo : 1330517224 download finish for photo 1330516957 begin to download photo : 1330516953 download finish for photo 1330517407 begin to download photo : 1330517405 download finish for photo 1330516680 begin to download photo : 1330514948 download finish for photo 1330516953 begin to download photo : 1330516937 download finish for photo 1330517224 begin to download photo : 1330517214 download finish for photo 1330517405 download finish for photo 1330514948 download finish for photo 1330516937 download finish for photo 1330517214 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19804580_05-02-2025_09_04_20.pdf results_Auto_P19804580_05-02-2025_09_04_20.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19804580_05-02-2025_09_04_20.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','19804580','results_Auto_P19804580_05-02-2025_09_04_20.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19804580_05-02-2025_09_04_20.pdf','pdf','','0.17','0.019060807214082234') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1330517409, 1330517407, 1330517405, 1330517303, 1330517301, 1330517228, 1330517224, 1330517214, 1330517210, 1330516960, 1330516957, 1330516953, 1330516937, 1330516688, 1330516685, 1330516682, 1330516680, 1330514948, 1330514942, 1330514941] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517409', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517407', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517405', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517303', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517301', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517228', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517224', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517214', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517210', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516960', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516957', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516953', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516937', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516688', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516685', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516682', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516680', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330514948', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330514942', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330514941', None, None, None, None, None, '2501423') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 20 time used for this insertion : 0.014138936996459961 save_final save missing photos in datou_result : time spend for datou_step_exec : 3.886476755142212 time spend to save output : 0.014443397521972656 total time spend for step 10 : 3.9009201526641846 step11:split_time_score Wed Feb 5 09:04:24 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('14', 28),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 21012025 19804580 Nombre de photos uploadées : 28 / 23040 (0%) 21012025 19804580 Nombre de photos taguées (types de déchets): 0 / 28 (0%) 21012025 19804580 Nombre de photos taguées (volume) : 0 / 28 (0%) elapsed_time : load_data_split_time_score 4.0531158447265625e-06 elapsed_time : order_list_meta_photo_and_scores 1.0013580322265625e-05 ???????????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0013282299041748047 elapsed_time : insert_dashboard_record_day_entry 0.022701025009155273 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.04809637498190257 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19791644_21-01-2025_09_02_16.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19791644 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19791644 AND mptpi.`type`=3726 To do Qualite : 0.0407800529489664 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19791645_21-01-2025_09_22_15.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19791645 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19791645 AND mptpi.`type`=3726 To do Qualite : 0.05806526819103342 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19792450_21-01-2025_08_51_18.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19792450 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19792450 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19795840 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19795842 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19795857 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19797270 order by id desc limit 1 Qualite : 0.0489462853096413 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19797271_21-01-2025_11_21_41.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19797271 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19797271 AND mptpi.`type`=3726 To do Qualite : 0.024754002648240084 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19797272_04-02-2025_12_00_16.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19797272 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19797272 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19798289 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19799584 order by id desc limit 1 Qualite : 0.034673920167750547 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19800775_04-02-2025_12_06_25.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19800775 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19800775 AND mptpi.`type`=3726 To do Qualite : 0.05428035426423572 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19801172_21-01-2025_13_33_11.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19801172 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19801172 AND mptpi.`type`=3726 To do Qualite : 0.029923338530732466 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19804578_05-02-2025_08_59_45.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19804578 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19804578 AND mptpi.`type`=3726 To do Qualite : 0.019499774534156192 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19804579_05-02-2025_09_00_39.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19804579 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19804579 AND mptpi.`type`=3726 To do Qualite : 0.019060807214082234 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19804580_05-02-2025_09_04_20.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19804580 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19804580 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19804581 order by id desc limit 1 Qualite : 0.03321823590790911 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19804718_04-02-2025_12_10_07.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19804718 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19804718 AND mptpi.`type`=3726 To do Qualite : 0.027324753035856044 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19806030_04-02-2025_12_10_38.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19806030 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19806030 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19806031 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19815786 order by id desc limit 1 Qualite : 0.038452217442877974 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19815787_04-02-2025_11_35_37.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19815787 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19815787 AND mptpi.`type`=3726 To do Qualite : 0.012731219344216792 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19815788_30-01-2025_12_41_32.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19815788 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19815788 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19815789 order by id desc limit 1 Qualite : 0.03828743062344873 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19815790_22-01-2025_12_21_28.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19815790 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19815790 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19815791 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19815792 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007294 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007296 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007297 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007298 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007299 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007300 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007301 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007302 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007303 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007304 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007305 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007306 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007307 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007308 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007309 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007334 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007359 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007360 order by id desc limit 1 Qualite : 0.021048126452339815 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20007361_28-01-2025_10_02_22.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007361 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20007361 AND mptpi.`type`=3726 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'21012025': {'nb_upload': 28, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1330517409, 1330517407, 1330517405, 1330517303, 1330517301, 1330517228, 1330517224, 1330517214, 1330517210, 1330516960, 1330516957, 1330516953, 1330516937, 1330516688, 1330516685, 1330516682, 1330516680, 1330514948, 1330514942, 1330514941] Looping around the photos to save general results len do output : 1 /19804580Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517409', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517407', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517405', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517303', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517301', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517228', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517224', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517214', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330517210', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516960', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516957', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516953', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516937', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516688', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516685', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516682', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330516680', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330514948', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330514942', None, None, None, None, None, '2501423') ('3459', None, None, None, None, None, None, None, '2501423') ('3459', '19804580', '1330514941', None, None, None, None, None, '2501423') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 21 time used for this insertion : 0.014948368072509766 save_final save missing photos in datou_result : time spend for datou_step_exec : 24.896972179412842 time spend to save output : 0.01525115966796875 total time spend for step 11 : 24.91222333908081 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 53.06user 22.61system 1:56.68elapsed 64%CPU (0avgtext+0avgdata 2955968maxresident)k 20600inputs+29632outputs (329major+1844363minor)pagefaults 0swaps