python /home/admin/mtr/script_for_cron.py -j default -m 20 -a 'python3 ~/workarea/git/Velours/python/prod/datou.py -j batch_current -C 2501421' -s traitement_3459 -M 0 -S 0 -U 100,80,95 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/home/admin/workarea/git/apy', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3069656 load datou : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : step 0 init_dummy_multi_datou is not linked in the step_by_step architecture ! WARNING : step 1294 init_dummy_multi_datou is not linked in the step_by_step architecture ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : (photo_id, hashtag_id, score_max) was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? load thcls load pdts Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2501421'] with mtr_portfolio_ids : ['19804578'] and first list_photo_ids : [] new path : /proc/3069656/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score over limit max, limiting to limit_max 20 list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 20 ; length of list_pids : 20 ; length of list_args : 20 time to download the photos : 6.700711488723755 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Wed Feb 5 08:58:10 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10998 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-05 08:58:13.647872: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-05 08:58:13.675228: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-05 08:58:13.677378: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f9cdc000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-05 08:58:13.677433: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-05 08:58:13.681268: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-05 08:58:13.988708: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x38afff60 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-05 08:58:13.988747: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-05 08:58:13.989777: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-05 08:58:13.990092: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-05 08:58:13.992287: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-05 08:58:13.994348: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-05 08:58:13.994663: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-05 08:58:13.996910: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-05 08:58:13.997911: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-05 08:58:14.002280: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-05 08:58:14.003853: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-05 08:58:14.003938: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-05 08:58:14.004755: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-05 08:58:14.004771: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-05 08:58:14.004795: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-05 08:58:14.006167: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10193 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-05 08:58:14.269069: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-05 08:58:14.269213: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-05 08:58:14.269246: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-05 08:58:14.269275: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-05 08:58:14.269304: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-05 08:58:14.269332: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-05 08:58:14.269359: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-05 08:58:14.269388: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-05 08:58:14.271568: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-05 08:58:14.272807: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-05 08:58:14.272836: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-05 08:58:14.272850: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-05 08:58:14.272863: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-05 08:58:14.272877: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-05 08:58:14.272890: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-05 08:58:14.272903: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-05 08:58:14.272915: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-05 08:58:14.274121: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-05 08:58:14.274155: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-05 08:58:14.274163: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-05 08:58:14.274170: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-05 08:58:14.275436: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10193 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-05 08:58:21.280368: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-05 08:58:21.456719: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 20 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 2 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 2 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 2 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 3 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 13 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 Detection mask done ! Trying to reset tf kernel 3069821 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5706 tf kernel not reseted sub process len(results) : 20 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 20 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 10842 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.0002903938293457031 nb_pixel_total : 10355 time to create 1 rle with old method : 0.010883331298828125 length of segment : 119 time for calcul the mask position with numpy : 0.018470287322998047 nb_pixel_total : 1038574 time to create 1 rle with new method : 0.03790163993835449 length of segment : 1817 time for calcul the mask position with numpy : 0.0035469532012939453 nb_pixel_total : 203707 time to create 1 rle with new method : 0.011057138442993164 length of segment : 971 time for calcul the mask position with numpy : 0.0011854171752929688 nb_pixel_total : 78431 time to create 1 rle with old method : 0.08216309547424316 length of segment : 390 time for calcul the mask position with numpy : 0.0008640289306640625 nb_pixel_total : 50835 time to create 1 rle with old method : 0.05324125289916992 length of segment : 217 time for calcul the mask position with numpy : 0.001004934310913086 nb_pixel_total : 64486 time to create 1 rle with old method : 0.0662226676940918 length of segment : 263 time for calcul the mask position with numpy : 0.00010085105895996094 nb_pixel_total : 3291 time to create 1 rle with old method : 0.0038945674896240234 length of segment : 70 time for calcul the mask position with numpy : 0.014722824096679688 nb_pixel_total : 962518 time to create 1 rle with new method : 0.031157493591308594 length of segment : 1296 time for calcul the mask position with numpy : 0.0014874935150146484 nb_pixel_total : 105462 time to create 1 rle with old method : 0.10843062400817871 length of segment : 381 time for calcul the mask position with numpy : 0.004208087921142578 nb_pixel_total : 262096 time to create 1 rle with new method : 0.00871729850769043 length of segment : 1187 time for calcul the mask position with numpy : 0.010686874389648438 nb_pixel_total : 538592 time to create 1 rle with new method : 0.03559136390686035 length of segment : 2368 time for calcul the mask position with numpy : 0.0051953792572021484 nb_pixel_total : 146583 time to create 1 rle with old method : 0.15684795379638672 length of segment : 659 time for calcul the mask position with numpy : 0.0007622241973876953 nb_pixel_total : 16314 time to create 1 rle with old method : 0.0177609920501709 length of segment : 323 time for calcul the mask position with numpy : 0.0005209445953369141 nb_pixel_total : 18692 time to create 1 rle with old method : 0.020621776580810547 length of segment : 148 time for calcul the mask position with numpy : 0.009285926818847656 nb_pixel_total : 299027 time to create 1 rle with new method : 0.01742100715637207 length of segment : 1063 time for calcul the mask position with numpy : 0.016299724578857422 nb_pixel_total : 819801 time to create 1 rle with new method : 0.031632423400878906 length of segment : 1806 time for calcul the mask position with numpy : 0.017534494400024414 nb_pixel_total : 939382 time to create 1 rle with new method : 0.03188753128051758 length of segment : 1811 time for calcul the mask position with numpy : 0.00018668174743652344 nb_pixel_total : 2273 time to create 1 rle with old method : 0.0025184154510498047 length of segment : 64 time for calcul the mask position with numpy : 0.005133867263793945 nb_pixel_total : 404887 time to create 1 rle with new method : 0.013959169387817383 length of segment : 1740 time for calcul the mask position with numpy : 0.022025585174560547 nb_pixel_total : 1260179 time to create 1 rle with new method : 0.034902334213256836 length of segment : 2238 time for calcul the mask position with numpy : 0.0001995563507080078 nb_pixel_total : 4242 time to create 1 rle with old method : 0.007378339767456055 length of segment : 61 time for calcul the mask position with numpy : 0.017520427703857422 nb_pixel_total : 1108647 time to create 1 rle with new method : 0.03381538391113281 length of segment : 1975 time for calcul the mask position with numpy : 0.00645136833190918 nb_pixel_total : 312221 time to create 1 rle with new method : 0.012568473815917969 length of segment : 1186 time for calcul the mask position with numpy : 0.001688241958618164 nb_pixel_total : 95572 time to create 1 rle with old method : 0.10478377342224121 length of segment : 632 time for calcul the mask position with numpy : 0.013676166534423828 nb_pixel_total : 837595 time to create 1 rle with new method : 0.03276419639587402 length of segment : 1132 time for calcul the mask position with numpy : 0.003876924514770508 nb_pixel_total : 328933 time to create 1 rle with new method : 0.009198665618896484 length of segment : 1034 time for calcul the mask position with numpy : 0.002267122268676758 nb_pixel_total : 151223 time to create 1 rle with new method : 0.005786895751953125 length of segment : 442 time for calcul the mask position with numpy : 0.001165628433227539 nb_pixel_total : 98551 time to create 1 rle with old method : 0.10799551010131836 length of segment : 441 time for calcul the mask position with numpy : 0.0040209293365478516 nb_pixel_total : 232722 time to create 1 rle with new method : 0.008379459381103516 length of segment : 507 time for calcul the mask position with numpy : 0.017368316650390625 nb_pixel_total : 1239583 time to create 1 rle with new method : 0.03560137748718262 length of segment : 1460 time for calcul the mask position with numpy : 0.0046024322509765625 nb_pixel_total : 319725 time to create 1 rle with new method : 0.010367155075073242 length of segment : 1020 time for calcul the mask position with numpy : 0.024419069290161133 nb_pixel_total : 1425955 time to create 1 rle with new method : 0.04268836975097656 length of segment : 1595 time for calcul the mask position with numpy : 0.0058057308197021484 nb_pixel_total : 359445 time to create 1 rle with new method : 0.017795324325561523 length of segment : 2261 time for calcul the mask position with numpy : 0.001135110855102539 nb_pixel_total : 43785 time to create 1 rle with old method : 0.04818916320800781 length of segment : 433 time for calcul the mask position with numpy : 0.002803802490234375 nb_pixel_total : 143773 time to create 1 rle with old method : 0.18339943885803223 length of segment : 404 time for calcul the mask position with numpy : 0.01212930679321289 nb_pixel_total : 769765 time to create 1 rle with new method : 0.028758525848388672 length of segment : 1291 time for calcul the mask position with numpy : 0.0016450881958007812 nb_pixel_total : 110598 time to create 1 rle with old method : 0.12198424339294434 length of segment : 513 time for calcul the mask position with numpy : 0.014759540557861328 nb_pixel_total : 1004632 time to create 1 rle with new method : 0.026352643966674805 length of segment : 1778 time for calcul the mask position with numpy : 0.004408597946166992 nb_pixel_total : 262213 time to create 1 rle with new method : 0.008938789367675781 length of segment : 918 time for calcul the mask position with numpy : 0.00532841682434082 nb_pixel_total : 322608 time to create 1 rle with new method : 0.012186765670776367 length of segment : 1066 time for calcul the mask position with numpy : 0.0005006790161132812 nb_pixel_total : 13980 time to create 1 rle with old method : 0.015732765197753906 length of segment : 204 time for calcul the mask position with numpy : 0.0014085769653320312 nb_pixel_total : 55688 time to create 1 rle with old method : 0.06180238723754883 length of segment : 380 time for calcul the mask position with numpy : 0.0028450489044189453 nb_pixel_total : 153377 time to create 1 rle with new method : 0.0055544376373291016 length of segment : 223 time for calcul the mask position with numpy : 0.00035071372985839844 nb_pixel_total : 9734 time to create 1 rle with old method : 0.013363122940063477 length of segment : 156 time for calcul the mask position with numpy : 0.0070340633392333984 nb_pixel_total : 55178 time to create 1 rle with old method : 0.06435537338256836 length of segment : 863 time for calcul the mask position with numpy : 0.0027434825897216797 nb_pixel_total : 64447 time to create 1 rle with old method : 0.07493042945861816 length of segment : 565 time for calcul the mask position with numpy : 0.0005123615264892578 nb_pixel_total : 38056 time to create 1 rle with old method : 0.04284191131591797 length of segment : 177 time for calcul the mask position with numpy : 0.0002982616424560547 nb_pixel_total : 7018 time to create 1 rle with old method : 0.008510828018188477 length of segment : 184 time for calcul the mask position with numpy : 0.007074117660522461 nb_pixel_total : 308888 time to create 1 rle with new method : 0.0178225040435791 length of segment : 904 time for calcul the mask position with numpy : 0.0038192272186279297 nb_pixel_total : 220370 time to create 1 rle with new method : 0.011485576629638672 length of segment : 1164 time for calcul the mask position with numpy : 0.003943204879760742 nb_pixel_total : 318690 time to create 1 rle with new method : 0.0093841552734375 length of segment : 1063 time for calcul the mask position with numpy : 0.0020415782928466797 nb_pixel_total : 155127 time to create 1 rle with new method : 0.004745960235595703 length of segment : 645 time for calcul the mask position with numpy : 0.004899263381958008 nb_pixel_total : 268262 time to create 1 rle with new method : 0.010264396667480469 length of segment : 1225 time for calcul the mask position with numpy : 0.009188413619995117 nb_pixel_total : 470950 time to create 1 rle with new method : 0.027690410614013672 length of segment : 1788 time for calcul the mask position with numpy : 0.003650188446044922 nb_pixel_total : 276997 time to create 1 rle with new method : 0.013156652450561523 length of segment : 1516 time for calcul the mask position with numpy : 0.00018405914306640625 nb_pixel_total : 2004 time to create 1 rle with old method : 0.002490997314453125 length of segment : 44 time for calcul the mask position with numpy : 0.01954054832458496 nb_pixel_total : 1202654 time to create 1 rle with new method : 0.03714394569396973 length of segment : 1826 time for calcul the mask position with numpy : 0.0037102699279785156 nb_pixel_total : 214317 time to create 1 rle with new method : 0.009818315505981445 length of segment : 1148 time for calcul the mask position with numpy : 0.0018329620361328125 nb_pixel_total : 60833 time to create 1 rle with old method : 0.07139968872070312 length of segment : 653 time for calcul the mask position with numpy : 0.0006644725799560547 nb_pixel_total : 27173 time to create 1 rle with old method : 0.03835892677307129 length of segment : 347 time for calcul the mask position with numpy : 0.0014324188232421875 nb_pixel_total : 46758 time to create 1 rle with old method : 0.056647539138793945 length of segment : 417 time for calcul the mask position with numpy : 0.0003991127014160156 nb_pixel_total : 9405 time to create 1 rle with old method : 0.011383533477783203 length of segment : 152 time for calcul the mask position with numpy : 0.017343521118164062 nb_pixel_total : 1033023 time to create 1 rle with new method : 0.03659319877624512 length of segment : 1192 time for calcul the mask position with numpy : 0.0014128684997558594 nb_pixel_total : 36700 time to create 1 rle with old method : 0.042351722717285156 length of segment : 245 time for calcul the mask position with numpy : 0.0002052783966064453 nb_pixel_total : 3509 time to create 1 rle with old method : 0.004461765289306641 length of segment : 44 time for calcul the mask position with numpy : 0.004074811935424805 nb_pixel_total : 140367 time to create 1 rle with old method : 0.15483522415161133 length of segment : 567 time for calcul the mask position with numpy : 0.00033736228942871094 nb_pixel_total : 8906 time to create 1 rle with old method : 0.010475635528564453 length of segment : 128 time for calcul the mask position with numpy : 0.0001728534698486328 nb_pixel_total : 2370 time to create 1 rle with old method : 0.002969503402709961 length of segment : 69 time for calcul the mask position with numpy : 0.01191091537475586 nb_pixel_total : 373209 time to create 1 rle with new method : 0.025511980056762695 length of segment : 928 time for calcul the mask position with numpy : 0.0014255046844482422 nb_pixel_total : 100525 time to create 1 rle with old method : 0.11031150817871094 length of segment : 467 time for calcul the mask position with numpy : 0.000423431396484375 nb_pixel_total : 12046 time to create 1 rle with old method : 0.014037370681762695 length of segment : 92 time for calcul the mask position with numpy : 0.0001971721649169922 nb_pixel_total : 3730 time to create 1 rle with old method : 0.0043909549713134766 length of segment : 113 time for calcul the mask position with numpy : 0.00530242919921875 nb_pixel_total : 395720 time to create 1 rle with new method : 0.012123823165893555 length of segment : 1522 time for calcul the mask position with numpy : 0.0001308917999267578 nb_pixel_total : 6227 time to create 1 rle with old method : 0.007067441940307617 length of segment : 110 time for calcul the mask position with numpy : 0.003911256790161133 nb_pixel_total : 188645 time to create 1 rle with new method : 0.005418062210083008 length of segment : 683 time for calcul the mask position with numpy : 0.014516353607177734 nb_pixel_total : 1154498 time to create 1 rle with new method : 0.02787327766418457 length of segment : 1680 time for calcul the mask position with numpy : 0.003941535949707031 nb_pixel_total : 316213 time to create 1 rle with new method : 0.010040521621704102 length of segment : 1000 time for calcul the mask position with numpy : 0.0003514289855957031 nb_pixel_total : 19396 time to create 1 rle with old method : 0.021947383880615234 length of segment : 199 time for calcul the mask position with numpy : 0.005994558334350586 nb_pixel_total : 293298 time to create 1 rle with new method : 0.01389455795288086 length of segment : 1348 time for calcul the mask position with numpy : 0.015600919723510742 nb_pixel_total : 1092642 time to create 1 rle with new method : 0.03497433662414551 length of segment : 2458 time for calcul the mask position with numpy : 0.0009138584136962891 nb_pixel_total : 63245 time to create 1 rle with old method : 0.06876063346862793 length of segment : 494 time for calcul the mask position with numpy : 0.0001373291015625 nb_pixel_total : 6721 time to create 1 rle with old method : 0.00812673568725586 length of segment : 101 time for calcul the mask position with numpy : 0.004085540771484375 nb_pixel_total : 326550 time to create 1 rle with new method : 0.01062464714050293 length of segment : 1399 time spent for convertir_results : 12.992801427841187 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 99 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Number RLEs to save : 69563 save missing photos in datou_result : time spend for datou_step_exec : 34.30419397354126 time spend to save output : 5.666733264923096 total time spend for step 1 : 39.970927238464355 step2:crop_condition Wed Feb 5 08:58:50 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 20 ! batch 1 Loaded 99 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1330511619, 1330511614, 1330511611, 1330511468, 1330511460, 1330511454, 1330511449, 1330511354, 1330511351, 1330511189, 1330511176, 1330511158, 1330511145, 1330511075, 1330511072, 1330510953, 1330510950, 1330510742, 1330510739, 1330510588] Looping around the photos to save general results len do output : 14 /-3627330798Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3627330845Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3627330797Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3627330796Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655540202Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3627330826Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655540203Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3627330841Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3627330847Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3627330851Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3627348303Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3627348292Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3627330843Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3627348299Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511619', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511614', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511611', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511468', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511460', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511454', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511449', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511354', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511351', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511189', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511176', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511158', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511145', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511075', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511072', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510953', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510950', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510742', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510739', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510588', None, None, None, None, None, '2501421') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 62 time used for this insertion : 0.8174183368682861 save_final save missing photos in datou_result : time spend for datou_step_exec : 6.633129358291626 time spend to save output : 0.8183126449584961 total time spend for step 2 : 7.451442003250122 step3:thcl Wed Feb 5 08:58:58 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.003309011459350586 time to convert the images to numpy array : 3.0994415283203125e-06 time to import caffe and check if the image exist : 0.004019737243652344 time to convert the images to numpy array : 1.6689300537109375e-06 time to import caffe and check if the image exist : 0.004123687744140625 time to convert the images to numpy array : 2.86102294921875e-06 time to import caffe and check if the image exist : 0.005307912826538086 time to convert the images to numpy array : 0.00807499885559082 time to import caffe and check if the image exist : 0.005361080169677734 time to convert the images to numpy array : 0.008741378784179688 time to import caffe and check if the image exist : 0.0057773590087890625 time to convert the images to numpy array : 0.021355867385864258 time to import caffe and check if the image exist : 0.006571531295776367 time to convert the images to numpy array : 0.020179271697998047 time to import caffe and check if the image exist : 0.004088640213012695 time to convert the images to numpy array : 0.027604103088378906 time to import caffe and check if the image exist : 0.005260467529296875 time to convert the images to numpy array : 0.02734375 time to import caffe and check if the image exist : 0.0040590763092041016 time to convert the images to numpy array : 0.032500267028808594 total time to convert the images to numpy array : 0.24851202964782715 list photo_ids error: [] list photo_ids correct : [-3627348303, -3627348292, -3627330843, -3627348299, -3627330797, -3627330796, -3627330847, -3627330851, -3627330798, -3627330845, -3655540203, -3627330841, -3655540202, -3627330826] number of photos to traite : 14 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 7218 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/:/home/admin/workarea/git/apy/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 6999 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.1545088291168213 time used to do the prediction : 0.08655858039855957 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 6997 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/:/home/admin/workarea/git/apy/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 6997 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.17200851440429688 time used to do the prediction : 0.0569462776184082 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 1.1205673217773438e-05 save missing photos in datou_result : time spend for datou_step_exec : 7.41320538520813 time spend to save output : 0.0003960132598876953 total time spend for step 3 : 7.413601398468018 step4:merge_mask_thcl_custom Wed Feb 5 08:59:05 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 99 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1330511619, 1330511614, 1330511611, 1330511468, 1330511460, 1330511454, 1330511449, 1330511354, 1330511351, 1330511189, 1330511176, 1330511158, 1330511145, 1330511075, 1330511072, 1330510953, 1330510950, 1330510742, 1330510739, 1330510588] Looping around the photos to save general results len do output : 20 /1330511619Didn't retrieve data .Didn't retrieve data . /1330511614Didn't retrieve data .Didn't retrieve data . /1330511611Didn't retrieve data .Didn't retrieve data . /1330511468Didn't retrieve data .Didn't retrieve data . /1330511460Didn't retrieve data .Didn't retrieve data . /1330511454Didn't retrieve data .Didn't retrieve data . /1330511449Didn't retrieve data .Didn't retrieve data . /1330511354Didn't retrieve data .Didn't retrieve data . /1330511351Didn't retrieve data .Didn't retrieve data . /1330511189Didn't retrieve data .Didn't retrieve data . /1330511176Didn't retrieve data .Didn't retrieve data . /1330511158Didn't retrieve data .Didn't retrieve data . /1330511145Didn't retrieve data .Didn't retrieve data . /1330511075Didn't retrieve data .Didn't retrieve data . /1330511072Didn't retrieve data .Didn't retrieve data . /1330510953Didn't retrieve data .Didn't retrieve data . /1330510950Didn't retrieve data .Didn't retrieve data . /1330510742Didn't retrieve data .Didn't retrieve data . /1330510739Didn't retrieve data .Didn't retrieve data . /1330510588Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511619', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511614', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511611', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511468', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511460', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511454', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511449', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511354', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511351', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511189', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511176', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511158', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511145', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511075', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511072', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510953', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510950', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510742', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510739', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510588', None, None, None, None, None, '2501421') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 60 time used for this insertion : 0.01689004898071289 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.06337261199951172 time spend to save output : 0.017440319061279297 total time spend for step 4 : 0.08081293106079102 step5:rle_unique_nms_with_priority Wed Feb 5 08:59:05 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 6 nb_hashtags : 3 time to prepare the origin masks : 0.7447748184204102 time for calcul the mask position with numpy : 0.013466596603393555 nb_pixel_total : 683645 time to create 1 rle with new method : 0.052961111068725586 time for calcul the mask position with numpy : 0.00644230842590332 nb_pixel_total : 77797 time to create 1 rle with old method : 0.07969093322753906 time for calcul the mask position with numpy : 0.007030010223388672 nb_pixel_total : 63596 time to create 1 rle with old method : 0.06479287147521973 time for calcul the mask position with numpy : 0.006908416748046875 nb_pixel_total : 152360 time to create 1 rle with new method : 0.04191708564758301 time for calcul the mask position with numpy : 0.015075922012329102 nb_pixel_total : 1035164 time to create 1 rle with new method : 0.04308271408081055 time for calcul the mask position with numpy : 0.0062427520751953125 nb_pixel_total : 50702 time to create 1 rle with old method : 0.07889008522033691 time for calcul the mask position with numpy : 0.006293773651123047 nb_pixel_total : 10336 time to create 1 rle with old method : 0.01144099235534668 create new chi : 0.44801902770996094 time to delete rle : 0.019879817962646484 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 7867 TO DO : save crop sub photo not yet done ! save time : 1.7649047374725342 nb_obj : 5 nb_hashtags : 1 time to prepare the origin masks : 0.48238182067871094 time for calcul the mask position with numpy : 0.01727008819580078 nb_pixel_total : 980639 time to create 1 rle with new method : 0.04275774955749512 time for calcul the mask position with numpy : 0.008173465728759766 nb_pixel_total : 80315 time to create 1 rle with old method : 0.08768868446350098 time for calcul the mask position with numpy : 0.006081104278564453 nb_pixel_total : 398 time to create 1 rle with old method : 0.0009214878082275391 time for calcul the mask position with numpy : 0.0061855316162109375 nb_pixel_total : 40664 time to create 1 rle with old method : 0.04326128959655762 time for calcul the mask position with numpy : 0.014935493469238281 nb_pixel_total : 968309 time to create 1 rle with new method : 0.02928614616394043 time for calcul the mask position with numpy : 0.006032705307006836 nb_pixel_total : 3275 time to create 1 rle with old method : 0.0036592483520507812 create new chi : 0.26720142364501953 time to delete rle : 0.00044655799865722656 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 5228 TO DO : save crop sub photo not yet done ! save time : 0.32427477836608887 nb_obj : 3 nb_hashtags : 1 time to prepare the origin masks : 0.29143619537353516 time for calcul the mask position with numpy : 0.017548561096191406 nb_pixel_total : 1498865 time to create 1 rle with new method : 0.0342259407043457 time for calcul the mask position with numpy : 0.006644010543823242 nb_pixel_total : 16245 time to create 1 rle with old method : 0.01748037338256836 time for calcul the mask position with numpy : 0.005942821502685547 nb_pixel_total : 21671 time to create 1 rle with old method : 0.02332139015197754 time for calcul the mask position with numpy : 0.010464906692504883 nb_pixel_total : 536819 time to create 1 rle with new method : 0.03334617614746094 create new chi : 0.14986085891723633 time to delete rle : 0.0006148815155029297 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 7265 TO DO : save crop sub photo not yet done ! save time : 0.5030117034912109 nb_obj : 4 nb_hashtags : 1 time to prepare the origin masks : 0.40631914138793945 time for calcul the mask position with numpy : 0.015926837921142578 nb_pixel_total : 1157823 time to create 1 rle with new method : 0.04727339744567871 time for calcul the mask position with numpy : 0.006438255310058594 nb_pixel_total : 1711 time to create 1 rle with old method : 0.003336668014526367 time for calcul the mask position with numpy : 0.012793302536010742 nb_pixel_total : 611501 time to create 1 rle with new method : 0.04743194580078125 time for calcul the mask position with numpy : 0.006529808044433594 nb_pixel_total : 3902 time to create 1 rle with old method : 0.005247354507446289 time for calcul the mask position with numpy : 0.008536577224731445 nb_pixel_total : 298663 time to create 1 rle with new method : 0.04638195037841797 create new chi : 0.20902419090270996 time to delete rle : 0.0013234615325927734 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 7902 TO DO : save crop sub photo not yet done ! save time : 0.4804871082305908 nb_obj : 4 nb_hashtags : 1 time to prepare the origin masks : 0.3949770927429199 time for calcul the mask position with numpy : 0.017189502716064453 nb_pixel_total : 971305 time to create 1 rle with new method : 0.044875144958496094 time for calcul the mask position with numpy : 0.008025169372558594 nb_pixel_total : 159276 time to create 1 rle with new method : 0.03611350059509277 time for calcul the mask position with numpy : 0.006113529205322266 nb_pixel_total : 2268 time to create 1 rle with old method : 0.0025959014892578125 time for calcul the mask position with numpy : 0.006078243255615234 nb_pixel_total : 4893 time to create 1 rle with old method : 0.0068628787994384766 time for calcul the mask position with numpy : 0.01217794418334961 nb_pixel_total : 935858 time to create 1 rle with new method : 0.03351306915283203 create new chi : 0.18093442916870117 time to delete rle : 0.0007836818695068359 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 7061 TO DO : save crop sub photo not yet done ! save time : 0.41126298904418945 nb_obj : 1 nb_hashtags : 1 time to prepare the origin masks : 0.29326939582824707 time for calcul the mask position with numpy : 0.013947248458862305 nb_pixel_total : 807307 time to create 1 rle with new method : 0.03887534141540527 time for calcul the mask position with numpy : 0.017773151397705078 nb_pixel_total : 1266293 time to create 1 rle with new method : 0.03431582450866699 create new chi : 0.1058812141418457 time to delete rle : 0.0008864402770996094 batch 1 Loaded 2 chid ids of type : 3726 Number RLEs to save : 5188 TO DO : save crop sub photo not yet done ! save time : 0.29781222343444824 nb_obj : 4 nb_hashtags : 1 time to prepare the origin masks : 0.665034294128418 time for calcul the mask position with numpy : 0.014252185821533203 nb_pixel_total : 798542 time to create 1 rle with new method : 0.04504823684692383 time for calcul the mask position with numpy : 0.006512880325317383 nb_pixel_total : 4194 time to create 1 rle with old method : 0.0051670074462890625 time for calcul the mask position with numpy : 0.007838249206542969 nb_pixel_total : 155656 time to create 1 rle with new method : 0.038553476333618164 time for calcul the mask position with numpy : 0.0072209835052490234 nb_pixel_total : 9190 time to create 1 rle with old method : 0.012159109115600586 time for calcul the mask position with numpy : 0.016550540924072266 nb_pixel_total : 1106018 time to create 1 rle with new method : 0.03772425651550293 create new chi : 0.19912004470825195 time to delete rle : 0.0009300708770751953 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 8813 TO DO : save crop sub photo not yet done ! save time : 0.6443979740142822 nb_obj : 5 nb_hashtags : 2 time to prepare the origin masks : 0.5116603374481201 time for calcul the mask position with numpy : 0.017138242721557617 nb_pixel_total : 894088 time to create 1 rle with new method : 0.04444599151611328 time for calcul the mask position with numpy : 0.0068471431732177734 nb_pixel_total : 150851 time to create 1 rle with new method : 0.03776431083679199 time for calcul the mask position with numpy : 0.006372928619384766 nb_pixel_total : 96901 time to create 1 rle with old method : 0.1011209487915039 time for calcul the mask position with numpy : 0.006237506866455078 nb_pixel_total : 58807 time to create 1 rle with old method : 0.061285972595214844 time for calcul the mask position with numpy : 0.0071790218353271484 nb_pixel_total : 30783 time to create 1 rle with old method : 0.03400015830993652 time for calcul the mask position with numpy : 0.013720989227294922 nb_pixel_total : 842170 time to create 1 rle with new method : 0.03470039367675781 create new chi : 0.3779575824737549 time to delete rle : 0.0007238388061523438 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 7671 TO DO : save crop sub photo not yet done ! save time : 0.814997673034668 nb_obj : 3 nb_hashtags : 1 time to prepare the origin masks : 0.3000049591064453 time for calcul the mask position with numpy : 0.017446517944335938 nb_pixel_total : 742012 time to create 1 rle with new method : 0.04675149917602539 time for calcul the mask position with numpy : 0.008296012878417969 nb_pixel_total : 44058 time to create 1 rle with old method : 0.04854178428649902 time for calcul the mask position with numpy : 0.017167329788208008 nb_pixel_total : 1055037 time to create 1 rle with new method : 0.0359499454498291 time for calcul the mask position with numpy : 0.007699728012084961 nb_pixel_total : 232493 time to create 1 rle with new method : 0.03521609306335449 create new chi : 0.21799445152282715 time to delete rle : 0.0005927085876464844 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 5268 TO DO : save crop sub photo not yet done ! save time : 0.4457881450653076 nb_obj : 2 nb_hashtags : 1 time to prepare the origin masks : 0.6389758586883545 time for calcul the mask position with numpy : 0.011760950088500977 nb_pixel_total : 588007 time to create 1 rle with new method : 0.04796862602233887 time for calcul the mask position with numpy : 0.008235931396484375 nb_pixel_total : 57417 time to create 1 rle with old method : 0.06264233589172363 time for calcul the mask position with numpy : 0.019328594207763672 nb_pixel_total : 1428176 time to create 1 rle with new method : 0.03946971893310547 create new chi : 0.1949446201324463 time to delete rle : 0.0004792213439941406 batch 1 Loaded 3 chid ids of type : 3726 Number RLEs to save : 4847 TO DO : save crop sub photo not yet done ! save time : 0.35881590843200684 nb_obj : 5 nb_hashtags : 2 time to prepare the origin masks : 0.6133561134338379 time for calcul the mask position with numpy : 0.019868850708007812 nb_pixel_total : 1116124 time to create 1 rle with new method : 0.04833412170410156 time for calcul the mask position with numpy : 0.006887912750244141 nb_pixel_total : 3266 time to create 1 rle with old method : 0.0039141178131103516 time for calcul the mask position with numpy : 0.00757908821105957 nb_pixel_total : 140559 time to create 1 rle with old method : 0.14710569381713867 time for calcul the mask position with numpy : 0.007676362991333008 nb_pixel_total : 4193 time to create 1 rle with old method : 0.005987882614135742 time for calcul the mask position with numpy : 0.013473749160766602 nb_pixel_total : 765692 time to create 1 rle with new method : 0.04689145088195801 time for calcul the mask position with numpy : 0.007014751434326172 nb_pixel_total : 43766 time to create 1 rle with old method : 0.04584622383117676 create new chi : 0.3693575859069824 time to delete rle : 0.0005819797515869141 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 6012 TO DO : save crop sub photo not yet done ! save time : 0.5086491107940674 nb_obj : 5 nb_hashtags : 1 time to prepare the origin masks : 0.6756191253662109 time for calcul the mask position with numpy : 0.01721811294555664 nb_pixel_total : 854860 time to create 1 rle with new method : 0.05086517333984375 time for calcul the mask position with numpy : 0.0066606998443603516 nb_pixel_total : 143 time to create 1 rle with old method : 0.0005288124084472656 time for calcul the mask position with numpy : 0.007945537567138672 nb_pixel_total : 106569 time to create 1 rle with old method : 0.11812973022460938 time for calcul the mask position with numpy : 0.0066683292388916016 nb_pixel_total : 97749 time to create 1 rle with old method : 0.10810494422912598 time for calcul the mask position with numpy : 0.006394624710083008 nb_pixel_total : 9281 time to create 1 rle with old method : 0.011668205261230469 time for calcul the mask position with numpy : 0.014284372329711914 nb_pixel_total : 1004998 time to create 1 rle with new method : 0.03935360908508301 create new chi : 0.39532899856567383 time to delete rle : 0.0006554126739501953 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 7516 TO DO : save crop sub photo not yet done ! save time : 0.43736958503723145 nb_obj : 6 nb_hashtags : 3 time to prepare the origin masks : 0.7270865440368652 time for calcul the mask position with numpy : 0.023060321807861328 nb_pixel_total : 1747537 time to create 1 rle with new method : 0.04070758819580078 time for calcul the mask position with numpy : 0.00625300407409668 nb_pixel_total : 55541 time to create 1 rle with old method : 0.059442996978759766 time for calcul the mask position with numpy : 0.006523609161376953 nb_pixel_total : 9680 time to create 1 rle with old method : 0.010323762893676758 time for calcul the mask position with numpy : 0.006497621536254883 nb_pixel_total : 33828 time to create 1 rle with old method : 0.03635287284851074 time for calcul the mask position with numpy : 0.006892681121826172 nb_pixel_total : 148925 time to create 1 rle with old method : 0.15343856811523438 time for calcul the mask position with numpy : 0.006057024002075195 nb_pixel_total : 13961 time to create 1 rle with old method : 0.014579296112060547 time for calcul the mask position with numpy : 0.006085395812988281 nb_pixel_total : 64128 time to create 1 rle with old method : 0.06594681739807129 create new chi : 0.44313812255859375 time to delete rle : 0.0007805824279785156 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 5545 TO DO : save crop sub photo not yet done ! save time : 0.40125489234924316 nb_obj : 6 nb_hashtags : 2 time to prepare the origin masks : 0.7346363067626953 time for calcul the mask position with numpy : 0.012724876403808594 nb_pixel_total : 1074763 time to create 1 rle with new method : 0.035338401794433594 time for calcul the mask position with numpy : 0.005862712860107422 nb_pixel_total : 6688 time to create 1 rle with old method : 0.00723719596862793 time for calcul the mask position with numpy : 0.008262157440185547 nb_pixel_total : 318545 time to create 1 rle with new method : 0.030397653579711914 time for calcul the mask position with numpy : 0.005826473236083984 nb_pixel_total : 1656 time to create 1 rle with old method : 0.0022547245025634766 time for calcul the mask position with numpy : 0.006691455841064453 nb_pixel_total : 207403 time to create 1 rle with new method : 0.027631521224975586 time for calcul the mask position with numpy : 0.007317781448364258 nb_pixel_total : 308861 time to create 1 rle with new method : 0.02794933319091797 time for calcul the mask position with numpy : 0.007359027862548828 nb_pixel_total : 155684 time to create 1 rle with new method : 0.027014970779418945 create new chi : 0.21542119979858398 time to delete rle : 0.0008244514465332031 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 9091 TO DO : save crop sub photo not yet done ! save time : 0.513310432434082 nb_obj : 4 nb_hashtags : 1 time to prepare the origin masks : 0.595782995223999 time for calcul the mask position with numpy : 0.015145301818847656 nb_pixel_total : 1048191 time to create 1 rle with new method : 0.03972673416137695 time for calcul the mask position with numpy : 0.0067446231842041016 nb_pixel_total : 172210 time to create 1 rle with new method : 0.03036332130432129 time for calcul the mask position with numpy : 0.006292819976806641 nb_pixel_total : 168122 time to create 1 rle with new method : 0.028268098831176758 time for calcul the mask position with numpy : 0.007914304733276367 nb_pixel_total : 417675 time to create 1 rle with new method : 0.028606176376342773 time for calcul the mask position with numpy : 0.006944417953491211 nb_pixel_total : 267402 time to create 1 rle with new method : 0.027643442153930664 create new chi : 0.20144081115722656 time to delete rle : 0.0008463859558105469 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 10732 TO DO : save crop sub photo not yet done ! save time : 0.757441520690918 nb_obj : 3 nb_hashtags : 1 time to prepare the origin masks : 0.41150403022766113 time for calcul the mask position with numpy : 0.013027191162109375 nb_pixel_total : 716189 time to create 1 rle with new method : 0.04576563835144043 time for calcul the mask position with numpy : 0.009430646896362305 nb_pixel_total : 43227 time to create 1 rle with old method : 0.047588348388671875 time for calcul the mask position with numpy : 0.0064697265625 nb_pixel_total : 113976 time to create 1 rle with old method : 0.11694979667663574 time for calcul the mask position with numpy : 0.01607370376586914 nb_pixel_total : 1200208 time to create 1 rle with new method : 0.03467249870300293 create new chi : 0.29819416999816895 time to delete rle : 0.0007219314575195312 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 6864 TO DO : save crop sub photo not yet done ! save time : 0.5460879802703857 nb_obj : 3 nb_hashtags : 1 time to prepare the origin masks : 0.28620457649230957 time for calcul the mask position with numpy : 0.014033794403076172 nb_pixel_total : 962940 time to create 1 rle with new method : 0.04538130760192871 time for calcul the mask position with numpy : 0.012772798538208008 nb_pixel_total : 1054679 time to create 1 rle with new method : 0.04028654098510742 time for calcul the mask position with numpy : 0.006243467330932617 nb_pixel_total : 9379 time to create 1 rle with old method : 0.010173320770263672 time for calcul the mask position with numpy : 0.006223201751708984 nb_pixel_total : 46602 time to create 1 rle with old method : 0.04879617691040039 create new chi : 0.19350981712341309 time to delete rle : 0.0004963874816894531 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 4290 TO DO : save crop sub photo not yet done ! save time : 0.25687098503112793 nb_obj : 11 nb_hashtags : 2 time to prepare the origin masks : 0.7084298133850098 time for calcul the mask position with numpy : 0.018959522247314453 nb_pixel_total : 1312717 time to create 1 rle with new method : 0.04380393028259277 time for calcul the mask position with numpy : 0.006720542907714844 nb_pixel_total : 168591 time to create 1 rle with new method : 0.0351557731628418 time for calcul the mask position with numpy : 0.005808353424072266 nb_pixel_total : 9908 time to create 1 rle with old method : 0.011034488677978516 time for calcul the mask position with numpy : 0.005859375 nb_pixel_total : 3720 time to create 1 rle with old method : 0.00394892692565918 time for calcul the mask position with numpy : 0.005827188491821289 nb_pixel_total : 13132 time to create 1 rle with old method : 0.014238595962524414 time for calcul the mask position with numpy : 0.005702972412109375 nb_pixel_total : 1677 time to create 1 rle with old method : 0.002382516860961914 time for calcul the mask position with numpy : 0.007703065872192383 nb_pixel_total : 372279 time to create 1 rle with new method : 0.0269012451171875 time for calcul the mask position with numpy : 0.006338357925415039 nb_pixel_total : 2363 time to create 1 rle with old method : 0.0027251243591308594 time for calcul the mask position with numpy : 0.0059010982513427734 nb_pixel_total : 8826 time to create 1 rle with old method : 0.009889602661132812 time for calcul the mask position with numpy : 0.006675004959106445 nb_pixel_total : 140285 time to create 1 rle with old method : 0.14379644393920898 time for calcul the mask position with numpy : 0.006072521209716797 nb_pixel_total : 3500 time to create 1 rle with old method : 0.0038208961486816406 time for calcul the mask position with numpy : 0.006010532379150391 nb_pixel_total : 36602 time to create 1 rle with old method : 0.038042545318603516 create new chi : 0.42894768714904785 time to delete rle : 0.0007336139678955078 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 7362 TO DO : save crop sub photo not yet done ! save time : 0.5338075160980225 nb_obj : 5 nb_hashtags : 2 time to prepare the origin masks : 0.5264458656311035 time for calcul the mask position with numpy : 0.010497331619262695 nb_pixel_total : 658019 time to create 1 rle with new method : 0.02961587905883789 time for calcul the mask position with numpy : 0.006159067153930664 nb_pixel_total : 19323 time to create 1 rle with old method : 0.02060723304748535 time for calcul the mask position with numpy : 0.005950927734375 nb_pixel_total : 55 time to create 1 rle with old method : 0.00011610984802246094 time for calcul the mask position with numpy : 0.006382465362548828 nb_pixel_total : 129890 time to create 1 rle with old method : 0.1385328769683838 time for calcul the mask position with numpy : 0.015333175659179688 nb_pixel_total : 1078185 time to create 1 rle with new method : 0.040990352630615234 time for calcul the mask position with numpy : 0.0069391727447509766 nb_pixel_total : 188128 time to create 1 rle with new method : 0.04148101806640625 create new chi : 0.3357846736907959 time to delete rle : 0.0005991458892822266 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 6529 TO DO : save crop sub photo not yet done ! save time : 0.5743472576141357 nb_obj : 4 nb_hashtags : 1 time to prepare the origin masks : 0.5207979679107666 time for calcul the mask position with numpy : 0.010323762893676758 nb_pixel_total : 571465 time to create 1 rle with new method : 0.033899784088134766 time for calcul the mask position with numpy : 0.010323762893676758 nb_pixel_total : 133319 time to create 1 rle with old method : 0.13689303398132324 time for calcul the mask position with numpy : 0.0064868927001953125 nb_pixel_total : 11728 time to create 1 rle with old method : 0.01277017593383789 time for calcul the mask position with numpy : 0.01423954963684082 nb_pixel_total : 1064701 time to create 1 rle with new method : 0.031204938888549805 time for calcul the mask position with numpy : 0.009014606475830078 nb_pixel_total : 292387 time to create 1 rle with new method : 0.027048826217651367 create new chi : 0.2930943965911865 time to delete rle : 0.0006330013275146484 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 7235 TO DO : save crop sub photo not yet done ! save time : 0.4160919189453125 map_output_result : {1330511619: (0.0, 'Should be the crop_list due to order', 0.0), 1330511614: (0.0, 'Should be the crop_list due to order', 0.0), 1330511611: (0.0, 'Should be the crop_list due to order', 0.0), 1330511468: (0.0, 'Should be the crop_list due to order', 0.0), 1330511460: (0.0, 'Should be the crop_list due to order', 0.0), 1330511454: (0.0, 'Should be the crop_list due to order', 0.0), 1330511449: (0.0, 'Should be the crop_list due to order', 0.0), 1330511354: (0.0, 'Should be the crop_list due to order', 0.0), 1330511351: (0.0, 'Should be the crop_list due to order', 0.0), 1330511189: (0.0, 'Should be the crop_list due to order', 0.0), 1330511176: (0.0, 'Should be the crop_list due to order', 0.0), 1330511158: (0.0, 'Should be the crop_list due to order', 0.0), 1330511145: (0.0, 'Should be the crop_list due to order', 0.0), 1330511075: (0.0, 'Should be the crop_list due to order', 0.0), 1330511072: (0.0, 'Should be the crop_list due to order', 0.0), 1330510953: (0.0, 'Should be the crop_list due to order', 0.0), 1330510950: (0.0, 'Should be the crop_list due to order', 0.0), 1330510742: (0.0, 'Should be the crop_list due to order', 0.0), 1330510739: (0.0, 'Should be the crop_list due to order', 0.0), 1330510588: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1330511619, 1330511614, 1330511611, 1330511468, 1330511460, 1330511454, 1330511449, 1330511354, 1330511351, 1330511189, 1330511176, 1330511158, 1330511145, 1330511075, 1330511072, 1330510953, 1330510950, 1330510742, 1330510739, 1330510588] Looping around the photos to save general results len do output : 20 /1330511619.Didn't retrieve data . /1330511614.Didn't retrieve data . /1330511611.Didn't retrieve data . /1330511468.Didn't retrieve data . /1330511460.Didn't retrieve data . /1330511454.Didn't retrieve data . /1330511449.Didn't retrieve data . /1330511354.Didn't retrieve data . /1330511351.Didn't retrieve data . /1330511189.Didn't retrieve data . /1330511176.Didn't retrieve data . /1330511158.Didn't retrieve data . /1330511145.Didn't retrieve data . /1330511075.Didn't retrieve data . /1330511072.Didn't retrieve data . /1330510953.Didn't retrieve data . /1330510950.Didn't retrieve data . /1330510742.Didn't retrieve data . /1330510739.Didn't retrieve data . /1330510588.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511619', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511614', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511611', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511468', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511460', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511454', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511449', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511354', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511351', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511189', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511176', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511158', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511145', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511075', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511072', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510953', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510950', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510742', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510739', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510588', None, None, None, None, None, '2501421') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 60 time used for this insertion : 0.014852762222290039 save_final save missing photos in datou_result : time spend for datou_step_exec : 28.40316128730774 time spend to save output : 0.015490293502807617 total time spend for step 5 : 28.418651580810547 step6:crop_condition Wed Feb 5 08:59:34 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 20 ! batch 1 Loaded 121 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738742376_3069656 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6406755447387695 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! About to upload 5 photos upload in portfolio : 4869462 init cache_photo without model_param we have 5 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738742378_3069656 we have uploaded 5 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.587738275527954 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738742380_3069656 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6166529655456543 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1330511619, 1330511614, 1330511611, 1330511468, 1330511460, 1330511454, 1330511449, 1330511354, 1330511351, 1330511189, 1330511176, 1330511158, 1330511145, 1330511075, 1330511072, 1330510953, 1330510950, 1330510742, 1330510739, 1330510588] Looping around the photos to save general results len do output : 7 /1334852810Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334852811Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334852812Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334852813Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334852814Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334852815Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334852816Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511619', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511614', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511611', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511468', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511460', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511454', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511449', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511354', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511351', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511189', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511176', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511158', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511145', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511075', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511072', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510953', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510950', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510742', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510739', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510588', None, None, None, None, None, '2501421') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 41 time used for this insertion : 0.02667713165283203 save_final save missing photos in datou_result : time spend for datou_step_exec : 7.729301929473877 time spend to save output : 0.027138948440551758 total time spend for step 6 : 7.756440877914429 step7:ventilate_hashtags_in_portfolio Wed Feb 5 08:59:42 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 19804578 get user id for portfolio 19804578 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19804578 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('cartonnette','mal_croppe','environnement','Carton_brun','plastique','autre_refus','metal','papier','flou','kraft','Teint_Dans_La_Masse','Carton_gris')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19804578 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('cartonnette','mal_croppe','environnement','Carton_brun','plastique','autre_refus','metal','papier','flou','kraft','Teint_Dans_La_Masse','Carton_gris')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19804578 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('cartonnette','mal_croppe','environnement','Carton_brun','plastique','autre_refus','metal','papier','flou','kraft','Teint_Dans_La_Masse','Carton_gris')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20248347,20248348,20248349,20248350,20248351,20248352,20248353,20248354,20248355,20248356,20248357,20248358?tags=cartonnette,mal_croppe,environnement,Carton_brun,plastique,autre_refus,metal,papier,flou,kraft,Teint_Dans_La_Masse,Carton_gris Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1330511619, 1330511614, 1330511611, 1330511468, 1330511460, 1330511454, 1330511449, 1330511354, 1330511351, 1330511189, 1330511176, 1330511158, 1330511145, 1330511075, 1330511072, 1330510953, 1330510950, 1330510742, 1330510739, 1330510588] Looping around the photos to save general results len do output : 1 /19804578. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511619', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511614', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511611', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511468', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511460', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511454', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511449', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511354', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511351', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511189', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511176', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511158', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511145', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511075', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511072', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510953', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510950', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510742', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510739', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510588', None, None, None, None, None, '2501421') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 21 time used for this insertion : 0.01574397087097168 save_final save missing photos in datou_result : time spend for datou_step_exec : 2.6367955207824707 time spend to save output : 0.01607036590576172 total time spend for step 7 : 2.6528658866882324 step8:final Wed Feb 5 08:59:44 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1330511619: ('0.029923338530732466',), 1330511614: ('0.029923338530732466',), 1330511611: ('0.029923338530732466',), 1330511468: ('0.029923338530732466',), 1330511460: ('0.029923338530732466',), 1330511454: ('0.029923338530732466',), 1330511449: ('0.029923338530732466',), 1330511354: ('0.029923338530732466',), 1330511351: ('0.029923338530732466',), 1330511189: ('0.029923338530732466',), 1330511176: ('0.029923338530732466',), 1330511158: ('0.029923338530732466',), 1330511145: ('0.029923338530732466',), 1330511075: ('0.029923338530732466',), 1330511072: ('0.029923338530732466',), 1330510953: ('0.029923338530732466',), 1330510950: ('0.029923338530732466',), 1330510742: ('0.029923338530732466',), 1330510739: ('0.029923338530732466',), 1330510588: ('0.029923338530732466',)} new output for save of step final : {1330511619: ('0.029923338530732466',), 1330511614: ('0.029923338530732466',), 1330511611: ('0.029923338530732466',), 1330511468: ('0.029923338530732466',), 1330511460: ('0.029923338530732466',), 1330511454: ('0.029923338530732466',), 1330511449: ('0.029923338530732466',), 1330511354: ('0.029923338530732466',), 1330511351: ('0.029923338530732466',), 1330511189: ('0.029923338530732466',), 1330511176: ('0.029923338530732466',), 1330511158: ('0.029923338530732466',), 1330511145: ('0.029923338530732466',), 1330511075: ('0.029923338530732466',), 1330511072: ('0.029923338530732466',), 1330510953: ('0.029923338530732466',), 1330510950: ('0.029923338530732466',), 1330510742: ('0.029923338530732466',), 1330510739: ('0.029923338530732466',), 1330510588: ('0.029923338530732466',)} [1330511619, 1330511614, 1330511611, 1330511468, 1330511460, 1330511454, 1330511449, 1330511354, 1330511351, 1330511189, 1330511176, 1330511158, 1330511145, 1330511075, 1330511072, 1330510953, 1330510950, 1330510742, 1330510739, 1330510588] Looping around the photos to save general results len do output : 20 /1330511619.Didn't retrieve data . /1330511614.Didn't retrieve data . /1330511611.Didn't retrieve data . /1330511468.Didn't retrieve data . /1330511460.Didn't retrieve data . /1330511454.Didn't retrieve data . /1330511449.Didn't retrieve data . /1330511354.Didn't retrieve data . /1330511351.Didn't retrieve data . /1330511189.Didn't retrieve data . /1330511176.Didn't retrieve data . /1330511158.Didn't retrieve data . /1330511145.Didn't retrieve data . /1330511075.Didn't retrieve data . /1330511072.Didn't retrieve data . /1330510953.Didn't retrieve data . /1330510950.Didn't retrieve data . /1330510742.Didn't retrieve data . /1330510739.Didn't retrieve data . /1330510588.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511619', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511614', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511611', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511468', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511460', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511454', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511449', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511354', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511351', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511189', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511176', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511158', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511145', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511075', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511072', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510953', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510950', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510742', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510739', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510588', None, None, None, None, None, '2501421') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 60 time used for this insertion : 0.014269590377807617 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.22087383270263672 time spend to save output : 0.015096902847290039 total time spend for step 8 : 0.23597073554992676 step9:velours_tree Wed Feb 5 08:59:44 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.029071331024169922 time spend to save output : 3.981590270996094e-05 total time spend for step 9 : 0.029111146926879883 step10:send_mail_cod Wed Feb 5 08:59:45 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin in order to get the selector url, please entre the license of selector results_Auto_P19804578_05-02-2025_08_59_45.pdf 20248347 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202483471738742385 20248348 imagette202483481738742385 20248351 imagette202483511738742385 20248352 change filename to text .imagette202483521738742385 20248353 imagette202483531738742385 20248355 imagette202483551738742385 20248356 imagette202483561738742385 20248350 change filename to text .imagette202483501738742385 20248357 imagette202483571738742385 20248358 imagette202483581738742385 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=19804578 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20248347,20248348,20248349,20248350,20248351,20248352,20248353,20248354,20248355,20248356,20248357,20248358?tags=cartonnette,mal_croppe,environnement,Carton_brun,plastique,autre_refus,metal,papier,flou,kraft,Teint_Dans_La_Masse,Carton_gris your option no_mail is active, we will not send the real mail to your client args[1330511619] : ('0.029923338530732466',) no score found for photo 1330511619 We are sending mail with results at report@fotonower.com args[1330511614] : ('0.029923338530732466',) no score found for photo 1330511614 We are sending mail with results at report@fotonower.com args[1330511611] : ('0.029923338530732466',) no score found for photo 1330511611 We are sending mail with results at report@fotonower.com args[1330511468] : ('0.029923338530732466',) no score found for photo 1330511468 We are sending mail with results at report@fotonower.com args[1330511460] : ('0.029923338530732466',) no score found for photo 1330511460 We are sending mail with results at report@fotonower.com args[1330511454] : ('0.029923338530732466',) no score found for photo 1330511454 We are sending mail with results at report@fotonower.com args[1330511449] : ('0.029923338530732466',) no score found for photo 1330511449 We are sending mail with results at report@fotonower.com args[1330511354] : ('0.029923338530732466',) no score found for photo 1330511354 We are sending mail with results at report@fotonower.com args[1330511351] : ('0.029923338530732466',) no score found for photo 1330511351 We are sending mail with results at report@fotonower.com args[1330511189] : ('0.029923338530732466',) no score found for photo 1330511189 We are sending mail with results at report@fotonower.com args[1330511176] : ('0.029923338530732466',) no score found for photo 1330511176 We are sending mail with results at report@fotonower.com args[1330511158] : ('0.029923338530732466',) no score found for photo 1330511158 We are sending mail with results at report@fotonower.com args[1330511145] : ('0.029923338530732466',) no score found for photo 1330511145 We are sending mail with results at report@fotonower.com args[1330511075] : ('0.029923338530732466',) no score found for photo 1330511075 We are sending mail with results at report@fotonower.com args[1330511072] : ('0.029923338530732466',) no score found for photo 1330511072 We are sending mail with results at report@fotonower.com args[1330510953] : ('0.029923338530732466',) no score found for photo 1330510953 We are sending mail with results at report@fotonower.com args[1330510950] : ('0.029923338530732466',) no score found for photo 1330510950 We are sending mail with results at report@fotonower.com args[1330510742] : ('0.029923338530732466',) no score found for photo 1330510742 We are sending mail with results at report@fotonower.com args[1330510739] : ('0.029923338530732466',) no score found for photo 1330510739 We are sending mail with results at report@fotonower.com args[1330510588] : ('0.029923338530732466',) no score found for photo 1330510588 We are sending mail with results at report@fotonower.com refus_total : 0.029923338530732466 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=19804578 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1330511072,1330511619,1330511614,1330511611,1330511468,1330511460,1330511454,1330511449,1330511354,1330511351,1330511189,1330511176,1330511158,1330511145,1330511075,1330509333,1330510953,1330510950,1330510742,1330510739) Found this number of photos: 20 begin to download photo : 1330511072 begin to download photo : 1330511460 begin to download photo : 1330511189 begin to download photo : 1330509333 download finish for photo 1330511189 begin to download photo : 1330511176 download finish for photo 1330511072 begin to download photo : 1330511619 download finish for photo 1330511460 begin to download photo : 1330511454 download finish for photo 1330509333 begin to download photo : 1330510953 download finish for photo 1330511176 begin to download photo : 1330511158 download finish for photo 1330511619 begin to download photo : 1330511614 download finish for photo 1330510953 begin to download photo : 1330510950 download finish for photo 1330511454 begin to download photo : 1330511449 download finish for photo 1330510950 begin to download photo : 1330510742 download finish for photo 1330511158 begin to download photo : 1330511145 download finish for photo 1330511614 begin to download photo : 1330511611 download finish for photo 1330511449 begin to download photo : 1330511354 download finish for photo 1330511145 begin to download photo : 1330511075 download finish for photo 1330511611 begin to download photo : 1330511468 download finish for photo 1330510742 begin to download photo : 1330510739 download finish for photo 1330511354 begin to download photo : 1330511351 download finish for photo 1330511075 download finish for photo 1330511351 download finish for photo 1330511468 download finish for photo 1330510739 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19804578_05-02-2025_08_59_45.pdf results_Auto_P19804578_05-02-2025_08_59_45.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19804578_05-02-2025_08_59_45.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','19804578','results_Auto_P19804578_05-02-2025_08_59_45.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19804578_05-02-2025_08_59_45.pdf','pdf','','0.23','0.029923338530732466') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1330511619, 1330511614, 1330511611, 1330511468, 1330511460, 1330511454, 1330511449, 1330511354, 1330511351, 1330511189, 1330511176, 1330511158, 1330511145, 1330511075, 1330511072, 1330510953, 1330510950, 1330510742, 1330510739, 1330510588] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511619', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511614', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511611', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511468', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511460', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511454', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511449', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511354', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511351', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511189', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511176', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511158', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511145', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511075', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511072', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510953', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510950', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510742', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510739', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510588', None, None, None, None, None, '2501421') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 20 time used for this insertion : 0.014734983444213867 save_final save missing photos in datou_result : time spend for datou_step_exec : 3.8740124702453613 time spend to save output : 0.014998674392700195 total time spend for step 10 : 3.8890111446380615 step11:split_time_score Wed Feb 5 08:59:48 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('13', 29),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 21012025 19804578 Nombre de photos uploadées : 29 / 23040 (0%) 21012025 19804578 Nombre de photos taguées (types de déchets): 0 / 29 (0%) 21012025 19804578 Nombre de photos taguées (volume) : 0 / 29 (0%) elapsed_time : load_data_split_time_score 2.384185791015625e-06 elapsed_time : order_list_meta_photo_and_scores 6.4373016357421875e-06 ????????????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0014581680297851562 elapsed_time : insert_dashboard_record_day_entry 0.02562093734741211 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.04809637498190257 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19791644_21-01-2025_09_02_16.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19791644 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19791644 AND mptpi.`type`=3726 To do Qualite : 0.0407800529489664 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19791645_21-01-2025_09_22_15.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19791645 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19791645 AND mptpi.`type`=3726 To do Qualite : 0.05806526819103342 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19792450_21-01-2025_08_51_18.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19792450 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19792450 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19795840 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19795842 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19795857 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19797270 order by id desc limit 1 Qualite : 0.0489462853096413 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19797271_21-01-2025_11_21_41.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19797271 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19797271 AND mptpi.`type`=3726 To do Qualite : 0.024754002648240084 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19797272_04-02-2025_12_00_16.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19797272 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19797272 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19798289 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19799584 order by id desc limit 1 Qualite : 0.034673920167750547 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19800775_04-02-2025_12_06_25.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19800775 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19800775 AND mptpi.`type`=3726 To do Qualite : 0.05428035426423572 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19801172_21-01-2025_13_33_11.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19801172 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19801172 AND mptpi.`type`=3726 To do Qualite : 0.029923338530732466 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19804578_05-02-2025_08_59_45.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19804578 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19804578 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19804579 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19804580 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19804581 order by id desc limit 1 Qualite : 0.03321823590790911 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19804718_04-02-2025_12_10_07.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19804718 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19804718 AND mptpi.`type`=3726 To do Qualite : 0.027324753035856044 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19806030_04-02-2025_12_10_38.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19806030 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19806030 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19806031 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19815786 order by id desc limit 1 Qualite : 0.038452217442877974 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19815787_04-02-2025_11_35_37.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19815787 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19815787 AND mptpi.`type`=3726 To do Qualite : 0.012731219344216792 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19815788_30-01-2025_12_41_32.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19815788 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19815788 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19815789 order by id desc limit 1 Qualite : 0.03828743062344873 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P19815790_22-01-2025_12_21_28.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19815790 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=19815790 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19815791 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 19815792 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007294 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007296 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007297 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007298 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007299 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007300 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007301 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007302 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007303 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007304 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007305 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007306 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007307 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007308 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007309 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007334 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007359 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007360 order by id desc limit 1 Qualite : 0.021048126452339815 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20007361_28-01-2025_10_02_22.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20007361 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20007361 AND mptpi.`type`=3726 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'21012025': {'nb_upload': 29, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1330511619, 1330511614, 1330511611, 1330511468, 1330511460, 1330511454, 1330511449, 1330511354, 1330511351, 1330511189, 1330511176, 1330511158, 1330511145, 1330511075, 1330511072, 1330510953, 1330510950, 1330510742, 1330510739, 1330510588] Looping around the photos to save general results len do output : 1 /19804578Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511619', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511614', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511611', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511468', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511460', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511454', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511449', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511354', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511351', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511189', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511176', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511158', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511145', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511075', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330511072', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510953', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510950', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510742', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510739', None, None, None, None, None, '2501421') ('3459', None, None, None, None, None, None, None, '2501421') ('3459', '19804578', '1330510588', None, None, None, None, None, '2501421') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 21 time used for this insertion : 0.015773296356201172 save_final save missing photos in datou_result : time spend for datou_step_exec : 32.40944957733154 time spend to save output : 0.015971660614013672 total time spend for step 11 : 32.42542123794556 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 60.07user 25.14system 2:20.12elapsed 60%CPU (0avgtext+0avgdata 2957960maxresident)k 576inputs+39560outputs (2major+1819036minor)pagefaults 0swaps