python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 20 -a ' -a 3318 ' -s datou_3318 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 313705 load datou : 3318 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5275, 'learn_RUBBIA_REFUS_AMIENS_23', 16384, 25088, 'learn_RUBBIA_REFUS_AMIENS_23', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 23, 14, 19, 39), datetime.datetime(2021, 4, 23, 14, 19, 39)) load thcls load THCL from format json or kwargs add thcl : 2847 in CacheModelConfig load pdts add pdt : 5275 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 list_input_json: [] Current got : datou_id : 3318, datou_cur_ids : ['2636519'] with mtr_portfolio_ids : ['21053304'] and first list_photo_ids : [] new path : /proc/313705/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, rle_unique_nms_with_priority, ventilate_hashtags_in_portfolio, final, blur_detection, brightness, velours_tree, send_mail_cod, split_time_score over limit max, limiting to limit_max 40 list_input_json : [] origin We have 1 , BFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 5 ; length of list_pids : 5 ; length of list_args : 5 time to download the photos : 1.4506490230560303 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 10 step1:mask_detect Wed Mar 5 10:00:35 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10151 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-03-05 10:00:39.310099: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-05 10:00:39.321198: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-05 10:00:39.323232: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f1438000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-05 10:00:39.323275: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-05 10:00:39.326828: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-05 10:00:39.573072: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x9c22d80 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-05 10:00:39.573171: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-05 10:00:39.574751: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-05 10:00:39.575393: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 10:00:39.580466: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 10:00:39.584895: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-05 10:00:39.585413: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-05 10:00:39.591085: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-05 10:00:39.593732: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-05 10:00:39.603417: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-05 10:00:39.605212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-05 10:00:39.605317: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 10:00:39.606243: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-05 10:00:39.606268: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-05 10:00:39.606281: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-05 10:00:39.607944: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9399 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-03-05 10:00:39.976337: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-05 10:00:39.976424: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 10:00:39.976443: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 10:00:39.976461: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-05 10:00:39.976479: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-05 10:00:39.976495: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-05 10:00:39.976513: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-05 10:00:39.976539: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-05 10:00:39.977885: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-05 10:00:39.979301: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-05 10:00:39.979366: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 10:00:39.979382: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 10:00:39.979397: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-05 10:00:39.979411: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-05 10:00:39.979425: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-05 10:00:39.979439: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-05 10:00:39.979453: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-05 10:00:39.980771: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-05 10:00:39.980811: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-05 10:00:39.980823: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-05 10:00:39.980832: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-05 10:00:39.982232: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9399 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2847 thcls : [{'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}] thcl {'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5275 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5275, 'learn_RUBBIA_REFUS_AMIENS_23', 16384, 25088, 'learn_RUBBIA_REFUS_AMIENS_23', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 23, 14, 19, 39), datetime.datetime(2021, 4, 23, 14, 19, 39)) {'thcl': {'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'], 'list_hashtags_csv': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'svm_hashtag_type_desc': 5275, 'photo_desc_type': 5275, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_RUBBIA_REFUS_AMIENS_23 NUM_CLASSES 9 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_RUBBIA_REFUS_AMIENS_23 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-03-05 10:00:53.250692: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 10:00:53.472266: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_RUBBIA_REFUS_AMIENS_23 /data/models_weight/learn_RUBBIA_REFUS_AMIENS_23/mask_model.h5 size_local : 256009536 size in s3 : 256009536 create time local : 2021-08-09 09:43:22 create time in s3 : 2021-08-06 18:54:04 mask_model.h5 already exist and didn't need to update list_images length : 5 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 148.85000 image_metas shape: (1, 17) min: 0.00000 max: 3840.00000 nb d'objets trouves : 32 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 147.10000 image_metas shape: (1, 17) min: 0.00000 max: 3840.00000 nb d'objets trouves : 59 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 149.85000 image_metas shape: (1, 17) min: 0.00000 max: 3840.00000 nb d'objets trouves : 53 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 146.85000 image_metas shape: (1, 17) min: 0.00000 max: 3840.00000 nb d'objets trouves : 56 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 3840.00000 nb d'objets trouves : 49 Detection mask done ! Trying to reset tf kernel 318404 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 4862 tf kernel not reseted sub process len(results) : 5 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 5 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 10372 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2847 Catched exception ! Connect or reconnect ! thcls : [{'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}] thcl {'id': 2847, 'mtr_user_id': 31, 'name': 'learn_RUBBIA_REFUS_AMIENS_23', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3594, 'photo_desc_type': 5275, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5275 ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'] time for calcul the mask position with numpy : 0.0009016990661621094 nb_pixel_total : 21897 time to create 1 rle with old method : 0.025017738342285156 length of segment : 154 time for calcul the mask position with numpy : 0.0004169940948486328 nb_pixel_total : 19279 time to create 1 rle with old method : 0.022762298583984375 length of segment : 182 time for calcul the mask position with numpy : 0.003228902816772461 nb_pixel_total : 111585 time to create 1 rle with old method : 0.12719202041625977 length of segment : 497 time for calcul the mask position with numpy : 0.0004661083221435547 nb_pixel_total : 17887 time to create 1 rle with old method : 0.021060466766357422 length of segment : 217 time for calcul the mask position with numpy : 0.0005750656127929688 nb_pixel_total : 27552 time to create 1 rle with old method : 0.03081822395324707 length of segment : 136 time for calcul the mask position with numpy : 0.0008561611175537109 nb_pixel_total : 31564 time to create 1 rle with old method : 0.03635668754577637 length of segment : 504 time for calcul the mask position with numpy : 0.00028824806213378906 nb_pixel_total : 11198 time to create 1 rle with old method : 0.012869834899902344 length of segment : 124 time for calcul the mask position with numpy : 0.0004096031188964844 nb_pixel_total : 20521 time to create 1 rle with old method : 0.023524045944213867 length of segment : 167 time for calcul the mask position with numpy : 0.0006220340728759766 nb_pixel_total : 20306 time to create 1 rle with old method : 0.023123741149902344 length of segment : 146 time for calcul the mask position with numpy : 0.0005571842193603516 nb_pixel_total : 26832 time to create 1 rle with old method : 0.030780553817749023 length of segment : 343 time for calcul the mask position with numpy : 0.00017189979553222656 nb_pixel_total : 6377 time to create 1 rle with old method : 0.007305145263671875 length of segment : 90 time for calcul the mask position with numpy : 0.0006251335144042969 nb_pixel_total : 14297 time to create 1 rle with old method : 0.016724824905395508 length of segment : 195 time for calcul the mask position with numpy : 0.005782127380371094 nb_pixel_total : 197236 time to create 1 rle with new method : 0.014786720275878906 length of segment : 593 time for calcul the mask position with numpy : 0.0006694793701171875 nb_pixel_total : 23526 time to create 1 rle with old method : 0.026216745376586914 length of segment : 240 time for calcul the mask position with numpy : 0.00031065940856933594 nb_pixel_total : 12562 time to create 1 rle with old method : 0.014780759811401367 length of segment : 123 time for calcul the mask position with numpy : 0.0005078315734863281 nb_pixel_total : 18553 time to create 1 rle with old method : 0.021238088607788086 length of segment : 191 time for calcul the mask position with numpy : 0.00015616416931152344 nb_pixel_total : 5321 time to create 1 rle with old method : 0.006369829177856445 length of segment : 114 time for calcul the mask position with numpy : 0.0007200241088867188 nb_pixel_total : 27902 time to create 1 rle with old method : 0.03153729438781738 length of segment : 175 time for calcul the mask position with numpy : 0.00041961669921875 nb_pixel_total : 16215 time to create 1 rle with old method : 0.018893003463745117 length of segment : 180 time for calcul the mask position with numpy : 0.00024199485778808594 nb_pixel_total : 12305 time to create 1 rle with old method : 0.01434016227722168 length of segment : 114 time for calcul the mask position with numpy : 0.0009455680847167969 nb_pixel_total : 43074 time to create 1 rle with old method : 0.04861593246459961 length of segment : 278 time for calcul the mask position with numpy : 0.0002377033233642578 nb_pixel_total : 7699 time to create 1 rle with old method : 0.008795022964477539 length of segment : 152 time for calcul the mask position with numpy : 0.0004298686981201172 nb_pixel_total : 21110 time to create 1 rle with old method : 0.02391958236694336 length of segment : 158 time for calcul the mask position with numpy : 0.00028228759765625 nb_pixel_total : 12410 time to create 1 rle with old method : 0.014602422714233398 length of segment : 153 time for calcul the mask position with numpy : 0.0002205371856689453 nb_pixel_total : 8591 time to create 1 rle with old method : 0.010280609130859375 length of segment : 96 time for calcul the mask position with numpy : 0.0006384849548339844 nb_pixel_total : 27118 time to create 1 rle with old method : 0.03051447868347168 length of segment : 214 time for calcul the mask position with numpy : 0.0012502670288085938 nb_pixel_total : 52452 time to create 1 rle with old method : 0.06020212173461914 length of segment : 313 time for calcul the mask position with numpy : 0.0026416778564453125 nb_pixel_total : 97734 time to create 1 rle with old method : 0.11179113388061523 length of segment : 480 time for calcul the mask position with numpy : 0.0004956722259521484 nb_pixel_total : 24119 time to create 1 rle with old method : 0.030879735946655273 length of segment : 202 time for calcul the mask position with numpy : 0.0002846717834472656 nb_pixel_total : 10873 time to create 1 rle with old method : 0.01247715950012207 length of segment : 199 time for calcul the mask position with numpy : 0.001333475112915039 nb_pixel_total : 22358 time to create 1 rle with old method : 0.025348186492919922 length of segment : 419 time for calcul the mask position with numpy : 0.0009481906890869141 nb_pixel_total : 50794 time to create 1 rle with old method : 0.05641007423400879 length of segment : 297 time for calcul the mask position with numpy : 0.00021004676818847656 nb_pixel_total : 9537 time to create 1 rle with old method : 0.016643762588500977 length of segment : 102 time for calcul the mask position with numpy : 0.00029921531677246094 nb_pixel_total : 13267 time to create 1 rle with old method : 0.015233755111694336 length of segment : 152 time for calcul the mask position with numpy : 0.0007686614990234375 nb_pixel_total : 38769 time to create 1 rle with old method : 0.04388022422790527 length of segment : 219 time for calcul the mask position with numpy : 0.00021696090698242188 nb_pixel_total : 8428 time to create 1 rle with old method : 0.010081052780151367 length of segment : 119 time for calcul the mask position with numpy : 0.00043272972106933594 nb_pixel_total : 18395 time to create 1 rle with old method : 0.021144390106201172 length of segment : 137 time for calcul the mask position with numpy : 0.0001804828643798828 nb_pixel_total : 7712 time to create 1 rle with old method : 0.009211540222167969 length of segment : 88 time for calcul the mask position with numpy : 0.000362396240234375 nb_pixel_total : 18081 time to create 1 rle with old method : 0.02131962776184082 length of segment : 139 time for calcul the mask position with numpy : 0.0006794929504394531 nb_pixel_total : 28430 time to create 1 rle with old method : 0.03307032585144043 length of segment : 282 time for calcul the mask position with numpy : 0.00017213821411132812 nb_pixel_total : 4741 time to create 1 rle with old method : 0.0056879520416259766 length of segment : 110 time for calcul the mask position with numpy : 0.00031065940856933594 nb_pixel_total : 12378 time to create 1 rle with old method : 0.014594793319702148 length of segment : 109 time for calcul the mask position with numpy : 0.0002415180206298828 nb_pixel_total : 10820 time to create 1 rle with old method : 0.013007640838623047 length of segment : 99 time for calcul the mask position with numpy : 0.00048613548278808594 nb_pixel_total : 8516 time to create 1 rle with old method : 0.009991168975830078 length of segment : 82 time for calcul the mask position with numpy : 0.0031697750091552734 nb_pixel_total : 189764 time to create 1 rle with new method : 0.012239933013916016 length of segment : 376 time for calcul the mask position with numpy : 0.0005085468292236328 nb_pixel_total : 11238 time to create 1 rle with old method : 0.013359546661376953 length of segment : 208 time for calcul the mask position with numpy : 0.00020432472229003906 nb_pixel_total : 7458 time to create 1 rle with old method : 0.00869894027709961 length of segment : 116 time for calcul the mask position with numpy : 0.0023467540740966797 nb_pixel_total : 55730 time to create 1 rle with old method : 0.06373333930969238 length of segment : 385 time for calcul the mask position with numpy : 0.0004956722259521484 nb_pixel_total : 15307 time to create 1 rle with old method : 0.01831221580505371 length of segment : 146 time for calcul the mask position with numpy : 0.007553577423095703 nb_pixel_total : 195888 time to create 1 rle with new method : 0.027957916259765625 length of segment : 540 time for calcul the mask position with numpy : 0.0009865760803222656 nb_pixel_total : 17435 time to create 1 rle with old method : 0.020326614379882812 length of segment : 216 time for calcul the mask position with numpy : 0.00154876708984375 nb_pixel_total : 25738 time to create 1 rle with old method : 0.029039859771728516 length of segment : 211 time for calcul the mask position with numpy : 0.0027604103088378906 nb_pixel_total : 57216 time to create 1 rle with old method : 0.0643620491027832 length of segment : 271 time for calcul the mask position with numpy : 0.002162933349609375 nb_pixel_total : 40401 time to create 1 rle with old method : 0.050188302993774414 length of segment : 275 time for calcul the mask position with numpy : 0.0007064342498779297 nb_pixel_total : 13891 time to create 1 rle with old method : 0.0166323184967041 length of segment : 145 time for calcul the mask position with numpy : 0.0003528594970703125 nb_pixel_total : 5734 time to create 1 rle with old method : 0.006749629974365234 length of segment : 94 time for calcul the mask position with numpy : 0.004582881927490234 nb_pixel_total : 108263 time to create 1 rle with old method : 0.12028002738952637 length of segment : 473 time for calcul the mask position with numpy : 0.0022144317626953125 nb_pixel_total : 50507 time to create 1 rle with old method : 0.059250831604003906 length of segment : 217 time for calcul the mask position with numpy : 0.0009202957153320312 nb_pixel_total : 14506 time to create 1 rle with old method : 0.02299952507019043 length of segment : 138 time for calcul the mask position with numpy : 0.0003349781036376953 nb_pixel_total : 4397 time to create 1 rle with old method : 0.007103919982910156 length of segment : 67 time for calcul the mask position with numpy : 0.0014247894287109375 nb_pixel_total : 28869 time to create 1 rle with old method : 0.0383756160736084 length of segment : 183 time for calcul the mask position with numpy : 0.0008924007415771484 nb_pixel_total : 10570 time to create 1 rle with old method : 0.012283086776733398 length of segment : 121 time for calcul the mask position with numpy : 0.0011243820190429688 nb_pixel_total : 10590 time to create 1 rle with old method : 0.01685500144958496 length of segment : 148 time for calcul the mask position with numpy : 0.0009770393371582031 nb_pixel_total : 14706 time to create 1 rle with old method : 0.02338886260986328 length of segment : 144 time for calcul the mask position with numpy : 0.0006730556488037109 nb_pixel_total : 10535 time to create 1 rle with old method : 0.01247715950012207 length of segment : 103 time for calcul the mask position with numpy : 0.01264190673828125 nb_pixel_total : 178268 time to create 1 rle with new method : 0.02041172981262207 length of segment : 667 time for calcul the mask position with numpy : 0.0015041828155517578 nb_pixel_total : 33624 time to create 1 rle with old method : 0.05015158653259277 length of segment : 322 time for calcul the mask position with numpy : 0.0004444122314453125 nb_pixel_total : 7062 time to create 1 rle with old method : 0.008530139923095703 length of segment : 70 time for calcul the mask position with numpy : 0.012120246887207031 nb_pixel_total : 320943 time to create 1 rle with new method : 0.018750905990600586 length of segment : 754 time for calcul the mask position with numpy : 0.0009903907775878906 nb_pixel_total : 19196 time to create 1 rle with old method : 0.02248239517211914 length of segment : 163 time for calcul the mask position with numpy : 0.00092315673828125 nb_pixel_total : 15790 time to create 1 rle with old method : 0.017974376678466797 length of segment : 139 time for calcul the mask position with numpy : 0.0011324882507324219 nb_pixel_total : 10311 time to create 1 rle with old method : 0.012410163879394531 length of segment : 256 time for calcul the mask position with numpy : 0.00040030479431152344 nb_pixel_total : 5466 time to create 1 rle with old method : 0.006494283676147461 length of segment : 90 time for calcul the mask position with numpy : 0.0008902549743652344 nb_pixel_total : 22644 time to create 1 rle with old method : 0.02780747413635254 length of segment : 101 time for calcul the mask position with numpy : 0.0018677711486816406 nb_pixel_total : 30732 time to create 1 rle with old method : 0.03686976432800293 length of segment : 189 time for calcul the mask position with numpy : 0.0017285346984863281 nb_pixel_total : 33305 time to create 1 rle with old method : 0.038442134857177734 length of segment : 177 time for calcul the mask position with numpy : 0.0015938282012939453 nb_pixel_total : 20306 time to create 1 rle with old method : 0.02420520782470703 length of segment : 176 time for calcul the mask position with numpy : 0.0057337284088134766 nb_pixel_total : 127878 time to create 1 rle with old method : 0.146195650100708 length of segment : 494 time for calcul the mask position with numpy : 0.0005257129669189453 nb_pixel_total : 9692 time to create 1 rle with old method : 0.011352777481079102 length of segment : 122 time for calcul the mask position with numpy : 0.0006129741668701172 nb_pixel_total : 11748 time to create 1 rle with old method : 0.013772726058959961 length of segment : 129 time for calcul the mask position with numpy : 0.0005922317504882812 nb_pixel_total : 10597 time to create 1 rle with old method : 0.012221813201904297 length of segment : 137 time for calcul the mask position with numpy : 0.00038433074951171875 nb_pixel_total : 7023 time to create 1 rle with old method : 0.008403301239013672 length of segment : 73 time for calcul the mask position with numpy : 0.00037980079650878906 nb_pixel_total : 5498 time to create 1 rle with old method : 0.006550788879394531 length of segment : 85 time for calcul the mask position with numpy : 0.002059459686279297 nb_pixel_total : 33113 time to create 1 rle with old method : 0.03807234764099121 length of segment : 230 time for calcul the mask position with numpy : 0.0017802715301513672 nb_pixel_total : 23981 time to create 1 rle with old method : 0.027722597122192383 length of segment : 374 time for calcul the mask position with numpy : 0.0007021427154541016 nb_pixel_total : 11670 time to create 1 rle with old method : 0.013550758361816406 length of segment : 107 time for calcul the mask position with numpy : 0.0015668869018554688 nb_pixel_total : 28001 time to create 1 rle with old method : 0.03408193588256836 length of segment : 263 time for calcul the mask position with numpy : 0.005032777786254883 nb_pixel_total : 109806 time to create 1 rle with old method : 0.1242673397064209 length of segment : 480 time for calcul the mask position with numpy : 0.0010476112365722656 nb_pixel_total : 17380 time to create 1 rle with old method : 0.02146434783935547 length of segment : 179 time for calcul the mask position with numpy : 0.0005450248718261719 nb_pixel_total : 8686 time to create 1 rle with old method : 0.010240316390991211 length of segment : 120 time for calcul the mask position with numpy : 0.0008258819580078125 nb_pixel_total : 16267 time to create 1 rle with old method : 0.018900632858276367 length of segment : 136 time for calcul the mask position with numpy : 0.0009000301361083984 nb_pixel_total : 6682 time to create 1 rle with old method : 0.008002758026123047 length of segment : 351 time for calcul the mask position with numpy : 0.0016541481018066406 nb_pixel_total : 26755 time to create 1 rle with old method : 0.030861854553222656 length of segment : 231 time for calcul the mask position with numpy : 0.0008807182312011719 nb_pixel_total : 21524 time to create 1 rle with old method : 0.02520132064819336 length of segment : 127 time for calcul the mask position with numpy : 0.0016570091247558594 nb_pixel_total : 23134 time to create 1 rle with old method : 0.026833772659301758 length of segment : 216 time for calcul the mask position with numpy : 0.0008413791656494141 nb_pixel_total : 13829 time to create 1 rle with old method : 0.0160825252532959 length of segment : 153 time for calcul the mask position with numpy : 0.0015265941619873047 nb_pixel_total : 22113 time to create 1 rle with old method : 0.026242733001708984 length of segment : 181 time for calcul the mask position with numpy : 0.0004277229309082031 nb_pixel_total : 8117 time to create 1 rle with old method : 0.009765148162841797 length of segment : 77 time for calcul the mask position with numpy : 0.0012264251708984375 nb_pixel_total : 23658 time to create 1 rle with old method : 0.027067899703979492 length of segment : 230 time for calcul the mask position with numpy : 0.0003681182861328125 nb_pixel_total : 6017 time to create 1 rle with old method : 0.007060527801513672 length of segment : 101 time for calcul the mask position with numpy : 0.0018351078033447266 nb_pixel_total : 11361 time to create 1 rle with old method : 0.014086484909057617 length of segment : 169 time for calcul the mask position with numpy : 0.0006384849548339844 nb_pixel_total : 11023 time to create 1 rle with old method : 0.012642145156860352 length of segment : 106 time for calcul the mask position with numpy : 0.0004744529724121094 nb_pixel_total : 9675 time to create 1 rle with old method : 0.011360883712768555 length of segment : 93 time for calcul the mask position with numpy : 0.007695913314819336 nb_pixel_total : 136166 time to create 1 rle with old method : 0.1546633243560791 length of segment : 774 time for calcul the mask position with numpy : 0.003353118896484375 nb_pixel_total : 58130 time to create 1 rle with old method : 0.07107901573181152 length of segment : 303 time for calcul the mask position with numpy : 0.0020551681518554688 nb_pixel_total : 44233 time to create 1 rle with old method : 0.052918434143066406 length of segment : 264 time for calcul the mask position with numpy : 0.0018322467803955078 nb_pixel_total : 35015 time to create 1 rle with old method : 0.039267778396606445 length of segment : 239 time spent for convertir_results : 6.908539533615112 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 107 chid ids of type : 3594 Number RLEs to save : 23309 save missing photos in datou_result : time spend for datou_step_exec : 78.29458260536194 time spend to save output : 3.578949213027954 total time spend for step 1 : 81.87353181838989 step2:crop_condition Wed Mar 5 10:01:57 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3594 Loading chi in step crop for list_pids : 5 ! batch 1 Loaded 107 chid ids of type : 3594 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : papier param for this class : {'min_score': 0.7} filtre for class : papier hashtag_id of this class : 492668766 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 75 About to insert : list_path_to_insert length 75 new photo from crops ! About to upload 75 photos upload in portfolio : 3736932 init cache_photo without model_param we have 75 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741165331_313705 we have uploaded 75 photos in the portfolio 3736932 time of upload the photos Elapsed time : 32.26127004623413 we have finished the crop for the class : papier begin to crop the class : carton param for this class : {'min_score': 0.7} filtre for class : carton hashtag_id of this class : 492774966 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 15 About to insert : list_path_to_insert length 15 new photo from crops ! About to upload 15 photos upload in portfolio : 3736932 init cache_photo without model_param we have 15 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741165368_313705 we have uploaded 15 photos in the portfolio 3736932 time of upload the photos Elapsed time : 4.770291090011597 we have finished the crop for the class : carton begin to crop the class : metal param for this class : {'min_score': 0.7} filtre for class : metal hashtag_id of this class : 492628673 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 3736932 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741165374_313705 we have uploaded 1 photos in the portfolio 3736932 time of upload the photos Elapsed time : 0.5679094791412354 we have finished the crop for the class : metal begin to crop the class : pet_clair param for this class : {'min_score': 0.7} filtre for class : pet_clair hashtag_id of this class : 2107755846 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 3736932 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741165378_313705 we have uploaded 6 photos in the portfolio 3736932 time of upload the photos Elapsed time : 2.1598238945007324 we have finished the crop for the class : pet_clair begin to crop the class : autre param for this class : {'min_score': 0.7} filtre for class : autre hashtag_id of this class : 494826614 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! About to upload 9 photos upload in portfolio : 3736932 init cache_photo without model_param we have 9 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741165382_313705 we have uploaded 9 photos in the portfolio 3736932 time of upload the photos Elapsed time : 3.805436134338379 we have finished the crop for the class : autre begin to crop the class : pehd param for this class : {'min_score': 0.7} filtre for class : pehd hashtag_id of this class : 628944319 begin to crop the class : pet_fonce param for this class : {'min_score': 0.7} filtre for class : pet_fonce hashtag_id of this class : 2107755900 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 3736932 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741165387_313705 we have uploaded 1 photos in the portfolio 3736932 time of upload the photos Elapsed time : 0.6479897499084473 we have finished the crop for the class : pet_fonce delete rles from all chi we have 0 chi objets contains the rles we have 0 chi objets contains the rles we have 0 chi objets contains the rles we have 0 chi objets contains the rles we have 0 chi objets contains the rles Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1334600578, 1334600573, 1334600570, 1334600563, 1334600552] Looping around the photos to save general results len do output : 107 /1340981009Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981010Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981011Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981012Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981014Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981015Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981016Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981017Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981018Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981019Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981021Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981023Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981025Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981027Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981031Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981034Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981035Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981036Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981037Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981038Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981039Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981040Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981041Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981042Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981043Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981044Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981045Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981046Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981047Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981048Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981049Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981050Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981052Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981053Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981054Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981055Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981056Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981057Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981058Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981059Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981060Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981061Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981062Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981064Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981066Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981067Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981068Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981069Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981070Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981071Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981072Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981073Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981074Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981075Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981076Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981077Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981078Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981079Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981081Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981083Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981084Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981085Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981087Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981088Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981089Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981090Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981091Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981092Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981093Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981094Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981095Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981096Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981097Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981099Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981100Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981134Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981136Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981138Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981140Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981142Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981143Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981144Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981145Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981146Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981147Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981148Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981150Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981151Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981152Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981153Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981155Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981158Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981159Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981160Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981161Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981162Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981163Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981166Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981167Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981168Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981169Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981170Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981171Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981173Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981175Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981177Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340981201Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600578', None, None, None, None, None, '2636519') ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600573', None, None, None, None, None, '2636519') ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600570', None, None, None, None, None, '2636519') ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600563', None, None, None, None, None, '2636519') ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600552', None, None, None, None, None, '2636519') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 326 time used for this insertion : 0.8827166557312012 save_final save missing photos in datou_result : time spend for datou_step_exec : 70.2122917175293 time spend to save output : 0.8875665664672852 total time spend for step 2 : 71.09985828399658 step3:rle_unique_nms_with_priority Wed Mar 5 10:03:08 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms batch 1 Loaded 107 chid ids of type : 3594 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++nb_obj : 11 nb_hashtags : 4 time to prepare the origin masks : 8.798089981079102 time for calcul the mask position with numpy : 1.2309060096740723 nb_pixel_total : 7979402 time to create 1 rle with new method : 1.091609001159668 time for calcul the mask position with numpy : 0.040845632553100586 nb_pixel_total : 6377 time to create 1 rle with old method : 0.0071599483489990234 time for calcul the mask position with numpy : 0.040761470794677734 nb_pixel_total : 26832 time to create 1 rle with old method : 0.03098297119140625 time for calcul the mask position with numpy : 0.04145216941833496 nb_pixel_total : 20306 time to create 1 rle with old method : 0.022133827209472656 time for calcul the mask position with numpy : 0.0417933464050293 nb_pixel_total : 20521 time to create 1 rle with old method : 0.02259969711303711 time for calcul the mask position with numpy : 0.04361534118652344 nb_pixel_total : 11198 time to create 1 rle with old method : 0.012524127960205078 time for calcul the mask position with numpy : 0.045928239822387695 nb_pixel_total : 31564 time to create 1 rle with old method : 0.03978395462036133 time for calcul the mask position with numpy : 0.043471336364746094 nb_pixel_total : 27552 time to create 1 rle with old method : 0.030175209045410156 time for calcul the mask position with numpy : 0.04168057441711426 nb_pixel_total : 17887 time to create 1 rle with old method : 0.020476818084716797 time for calcul the mask position with numpy : 0.04178309440612793 nb_pixel_total : 111585 time to create 1 rle with old method : 0.142822265625 time for calcul the mask position with numpy : 0.04296231269836426 nb_pixel_total : 19279 time to create 1 rle with old method : 0.029397010803222656 time for calcul the mask position with numpy : 0.06483936309814453 nb_pixel_total : 21897 time to create 1 rle with old method : 0.024461746215820312 create new chi : 3.239499807357788 time to delete rle : 0.02380657196044922 batch 1 Loaded 23 chid ids of type : 3594 +++++++++++++++Number RLEs to save : 7280 TO DO : save crop sub photo not yet done ! save time : 1.0127780437469482 nb_obj : 31 nb_hashtags : 3 time to prepare the origin masks : 4.696527004241943 time for calcul the mask position with numpy : 0.8114335536956787 nb_pixel_total : 7428580 time to create 1 rle with new method : 0.6794900894165039 time for calcul the mask position with numpy : 0.035959482192993164 nb_pixel_total : 27902 time to create 1 rle with old method : 0.03152585029602051 time for calcul the mask position with numpy : 0.03689289093017578 nb_pixel_total : 21110 time to create 1 rle with old method : 0.024688243865966797 time for calcul the mask position with numpy : 0.03627419471740723 nb_pixel_total : 28430 time to create 1 rle with old method : 0.03332805633544922 time for calcul the mask position with numpy : 0.03700995445251465 nb_pixel_total : 8428 time to create 1 rle with old method : 0.010090827941894531 time for calcul the mask position with numpy : 0.036049604415893555 nb_pixel_total : 52452 time to create 1 rle with old method : 0.07403230667114258 time for calcul the mask position with numpy : 0.03874516487121582 nb_pixel_total : 18553 time to create 1 rle with old method : 0.022492408752441406 time for calcul the mask position with numpy : 0.03441643714904785 nb_pixel_total : 22358 time to create 1 rle with old method : 0.02455878257751465 time for calcul the mask position with numpy : 0.03469586372375488 nb_pixel_total : 43074 time to create 1 rle with old method : 0.04740762710571289 time for calcul the mask position with numpy : 0.03484702110290527 nb_pixel_total : 24119 time to create 1 rle with old method : 0.026463747024536133 time for calcul the mask position with numpy : 0.03567838668823242 nb_pixel_total : 97567 time to create 1 rle with old method : 0.10700488090515137 time for calcul the mask position with numpy : 0.03444218635559082 nb_pixel_total : 50794 time to create 1 rle with old method : 0.05559229850769043 time for calcul the mask position with numpy : 0.03463101387023926 nb_pixel_total : 13267 time to create 1 rle with old method : 0.015897274017333984 time for calcul the mask position with numpy : 0.034807443618774414 nb_pixel_total : 18395 time to create 1 rle with old method : 0.021318435668945312 time for calcul the mask position with numpy : 0.036704301834106445 nb_pixel_total : 197236 time to create 1 rle with new method : 0.6493861675262451 time for calcul the mask position with numpy : 0.03529620170593262 nb_pixel_total : 27118 time to create 1 rle with old method : 0.0335698127746582 time for calcul the mask position with numpy : 0.0381319522857666 nb_pixel_total : 38769 time to create 1 rle with old method : 0.04261374473571777 time for calcul the mask position with numpy : 0.03477978706359863 nb_pixel_total : 7699 time to create 1 rle with old method : 0.009594917297363281 time for calcul the mask position with numpy : 0.038007259368896484 nb_pixel_total : 12378 time to create 1 rle with old method : 0.013670921325683594 time for calcul the mask position with numpy : 0.034993648529052734 nb_pixel_total : 23526 time to create 1 rle with old method : 0.026050567626953125 time for calcul the mask position with numpy : 0.03400993347167969 nb_pixel_total : 12410 time to create 1 rle with old method : 0.013491392135620117 time for calcul the mask position with numpy : 0.03451967239379883 nb_pixel_total : 14297 time to create 1 rle with old method : 0.015977144241333008 time for calcul the mask position with numpy : 0.03465676307678223 nb_pixel_total : 4741 time to create 1 rle with old method : 0.005399942398071289 time for calcul the mask position with numpy : 0.034691810607910156 nb_pixel_total : 16215 time to create 1 rle with old method : 0.01785135269165039 time for calcul the mask position with numpy : 0.03491616249084473 nb_pixel_total : 10873 time to create 1 rle with old method : 0.011943578720092773 time for calcul the mask position with numpy : 0.03472471237182617 nb_pixel_total : 18081 time to create 1 rle with old method : 0.020218372344970703 time for calcul the mask position with numpy : 0.034874916076660156 nb_pixel_total : 12562 time to create 1 rle with old method : 0.013948440551757812 time for calcul the mask position with numpy : 0.03507041931152344 nb_pixel_total : 12305 time to create 1 rle with old method : 0.013853073120117188 time for calcul the mask position with numpy : 0.03466439247131348 nb_pixel_total : 5321 time to create 1 rle with old method : 0.005932331085205078 time for calcul the mask position with numpy : 0.035860300064086914 nb_pixel_total : 9537 time to create 1 rle with old method : 0.010772943496704102 time for calcul the mask position with numpy : 0.03833723068237305 nb_pixel_total : 8591 time to create 1 rle with old method : 0.010101556777954102 time for calcul the mask position with numpy : 0.03647947311401367 nb_pixel_total : 7712 time to create 1 rle with old method : 0.008549928665161133 create new chi : 4.090799808502197 time to delete rle : 0.0023272037506103516 batch 1 Loaded 63 chid ids of type : 3594 ++++++++++++++++++++++++++++++++++++++++Number RLEs to save : 14830 TO DO : save crop sub photo not yet done ! save time : 4.924959182739258 nb_obj : 22 nb_hashtags : 5 time to prepare the origin masks : 13.682469606399536 time for calcul the mask position with numpy : 0.3429274559020996 nb_pixel_total : 7397062 time to create 1 rle with new method : 1.1398978233337402 time for calcul the mask position with numpy : 0.04248785972595215 nb_pixel_total : 14706 time to create 1 rle with old method : 0.01729297637939453 time for calcul the mask position with numpy : 0.042282819747924805 nb_pixel_total : 10590 time to create 1 rle with old method : 0.01331472396850586 time for calcul the mask position with numpy : 0.050201416015625 nb_pixel_total : 10570 time to create 1 rle with old method : 0.013782024383544922 time for calcul the mask position with numpy : 0.04951930046081543 nb_pixel_total : 28869 time to create 1 rle with old method : 0.03632473945617676 time for calcul the mask position with numpy : 0.044648170471191406 nb_pixel_total : 4397 time to create 1 rle with old method : 0.005078554153442383 time for calcul the mask position with numpy : 0.04189705848693848 nb_pixel_total : 14506 time to create 1 rle with old method : 0.01607370376586914 time for calcul the mask position with numpy : 0.040659189224243164 nb_pixel_total : 50507 time to create 1 rle with old method : 0.055086374282836914 time for calcul the mask position with numpy : 0.04092979431152344 nb_pixel_total : 108057 time to create 1 rle with old method : 0.11926531791687012 time for calcul the mask position with numpy : 0.04334568977355957 nb_pixel_total : 5734 time to create 1 rle with old method : 0.0063745975494384766 time for calcul the mask position with numpy : 0.04510068893432617 nb_pixel_total : 13891 time to create 1 rle with old method : 0.022549152374267578 time for calcul the mask position with numpy : 0.04257464408874512 nb_pixel_total : 40401 time to create 1 rle with old method : 0.04411745071411133 time for calcul the mask position with numpy : 0.04134202003479004 nb_pixel_total : 57216 time to create 1 rle with old method : 0.0687565803527832 time for calcul the mask position with numpy : 0.04509377479553223 nb_pixel_total : 25738 time to create 1 rle with old method : 0.03102731704711914 time for calcul the mask position with numpy : 0.04452180862426758 nb_pixel_total : 17435 time to create 1 rle with old method : 0.01995372772216797 time for calcul the mask position with numpy : 0.04721999168395996 nb_pixel_total : 195888 time to create 1 rle with new method : 0.7744300365447998 time for calcul the mask position with numpy : 0.04323172569274902 nb_pixel_total : 15307 time to create 1 rle with old method : 0.017231464385986328 time for calcul the mask position with numpy : 0.04169178009033203 nb_pixel_total : 55730 time to create 1 rle with old method : 0.06194901466369629 time for calcul the mask position with numpy : 0.03483295440673828 nb_pixel_total : 7458 time to create 1 rle with old method : 0.008684635162353516 time for calcul the mask position with numpy : 0.04019808769226074 nb_pixel_total : 11238 time to create 1 rle with old method : 0.012580156326293945 time for calcul the mask position with numpy : 0.03653883934020996 nb_pixel_total : 189764 time to create 1 rle with new method : 0.8899304866790771 time for calcul the mask position with numpy : 0.036432743072509766 nb_pixel_total : 8516 time to create 1 rle with old method : 0.009768009185791016 time for calcul the mask position with numpy : 0.03869152069091797 nb_pixel_total : 10820 time to create 1 rle with old method : 0.011818885803222656 create new chi : 4.777276277542114 time to delete rle : 0.0026199817657470703 batch 1 Loaded 45 chid ids of type : 3594 +++++++++++++++++++++++++Number RLEs to save : 11423 TO DO : save crop sub photo not yet done ! save time : 5.789466381072998 nb_obj : 18 nb_hashtags : 5 time to prepare the origin masks : 12.028453588485718 time for calcul the mask position with numpy : 1.2449290752410889 nb_pixel_total : 7430521 time to create 1 rle with new method : 1.9654104709625244 time for calcul the mask position with numpy : 0.045499563217163086 nb_pixel_total : 7023 time to create 1 rle with old method : 0.008118391036987305 time for calcul the mask position with numpy : 0.044370174407958984 nb_pixel_total : 10597 time to create 1 rle with old method : 0.011764287948608398 time for calcul the mask position with numpy : 0.04218745231628418 nb_pixel_total : 11748 time to create 1 rle with old method : 0.013616323471069336 time for calcul the mask position with numpy : 0.043233394622802734 nb_pixel_total : 9692 time to create 1 rle with old method : 0.010779142379760742 time for calcul the mask position with numpy : 0.04476284980773926 nb_pixel_total : 127878 time to create 1 rle with old method : 0.15150952339172363 time for calcul the mask position with numpy : 0.04348278045654297 nb_pixel_total : 20306 time to create 1 rle with old method : 0.022433996200561523 time for calcul the mask position with numpy : 0.043950796127319336 nb_pixel_total : 33305 time to create 1 rle with old method : 0.037865638732910156 time for calcul the mask position with numpy : 0.04243016242980957 nb_pixel_total : 30732 time to create 1 rle with old method : 0.03597426414489746 time for calcul the mask position with numpy : 0.050336599349975586 nb_pixel_total : 22644 time to create 1 rle with old method : 0.03638648986816406 time for calcul the mask position with numpy : 0.04860854148864746 nb_pixel_total : 5466 time to create 1 rle with old method : 0.006124973297119141 time for calcul the mask position with numpy : 0.04742908477783203 nb_pixel_total : 10311 time to create 1 rle with old method : 0.01365804672241211 time for calcul the mask position with numpy : 0.04442715644836426 nb_pixel_total : 15790 time to create 1 rle with old method : 0.019661426544189453 time for calcul the mask position with numpy : 0.04899907112121582 nb_pixel_total : 19196 time to create 1 rle with old method : 0.03049445152282715 time for calcul the mask position with numpy : 0.04946303367614746 nb_pixel_total : 320943 time to create 1 rle with new method : 1.0112802982330322 time for calcul the mask position with numpy : 0.043117523193359375 nb_pixel_total : 7062 time to create 1 rle with old method : 0.01096034049987793 time for calcul the mask position with numpy : 0.05041098594665527 nb_pixel_total : 22383 time to create 1 rle with old method : 0.030733585357666016 time for calcul the mask position with numpy : 0.04375624656677246 nb_pixel_total : 178268 time to create 1 rle with new method : 0.5710413455963135 time for calcul the mask position with numpy : 0.04133152961730957 nb_pixel_total : 10535 time to create 1 rle with old method : 0.012229681015014648 create new chi : 6.181155204772949 time to delete rle : 0.0030024051666259766 batch 1 Loaded 37 chid ids of type : 3594 +++++++++++++++++++++++Number RLEs to save : 10316 TO DO : save crop sub photo not yet done ! save time : 0.7561147212982178 nb_obj : 25 nb_hashtags : 3 time to prepare the origin masks : 16.047505140304565 time for calcul the mask position with numpy : 1.493237018585205 nb_pixel_total : 7582566 time to create 1 rle with new method : 0.5712766647338867 time for calcul the mask position with numpy : 0.04650449752807617 nb_pixel_total : 35015 time to create 1 rle with old method : 0.040038347244262695 time for calcul the mask position with numpy : 0.07868814468383789 nb_pixel_total : 44233 time to create 1 rle with old method : 0.050978660583496094 time for calcul the mask position with numpy : 0.0495457649230957 nb_pixel_total : 58130 time to create 1 rle with old method : 0.07724714279174805 time for calcul the mask position with numpy : 0.04568076133728027 nb_pixel_total : 136166 time to create 1 rle with old method : 0.15677380561828613 time for calcul the mask position with numpy : 0.04515504837036133 nb_pixel_total : 9675 time to create 1 rle with old method : 0.010695695877075195 time for calcul the mask position with numpy : 0.04673147201538086 nb_pixel_total : 11023 time to create 1 rle with old method : 0.012765884399414062 time for calcul the mask position with numpy : 0.04860877990722656 nb_pixel_total : 11361 time to create 1 rle with old method : 0.013214349746704102 time for calcul the mask position with numpy : 0.047063350677490234 nb_pixel_total : 6017 time to create 1 rle with old method : 0.009560346603393555 time for calcul the mask position with numpy : 0.04440593719482422 nb_pixel_total : 23658 time to create 1 rle with old method : 0.026105880737304688 time for calcul the mask position with numpy : 0.045324087142944336 nb_pixel_total : 8117 time to create 1 rle with old method : 0.00935053825378418 time for calcul the mask position with numpy : 0.04746842384338379 nb_pixel_total : 22113 time to create 1 rle with old method : 0.024257898330688477 time for calcul the mask position with numpy : 0.04375052452087402 nb_pixel_total : 13829 time to create 1 rle with old method : 0.015265464782714844 time for calcul the mask position with numpy : 0.044356346130371094 nb_pixel_total : 23134 time to create 1 rle with old method : 0.02548813819885254 time for calcul the mask position with numpy : 0.0442197322845459 nb_pixel_total : 21524 time to create 1 rle with old method : 0.023391008377075195 time for calcul the mask position with numpy : 0.04639148712158203 nb_pixel_total : 26755 time to create 1 rle with old method : 0.03266096115112305 time for calcul the mask position with numpy : 0.046030282974243164 nb_pixel_total : 6682 time to create 1 rle with old method : 0.0075778961181640625 time for calcul the mask position with numpy : 0.04460287094116211 nb_pixel_total : 16267 time to create 1 rle with old method : 0.019006013870239258 time for calcul the mask position with numpy : 0.04568076133728027 nb_pixel_total : 8686 time to create 1 rle with old method : 0.00958395004272461 time for calcul the mask position with numpy : 0.030080318450927734 nb_pixel_total : 17380 time to create 1 rle with old method : 0.020257949829101562 time for calcul the mask position with numpy : 0.0303192138671875 nb_pixel_total : 109806 time to create 1 rle with old method : 0.12061500549316406 time for calcul the mask position with numpy : 0.030824899673461914 nb_pixel_total : 28001 time to create 1 rle with old method : 0.031378984451293945 time for calcul the mask position with numpy : 0.03020763397216797 nb_pixel_total : 11670 time to create 1 rle with old method : 0.013412714004516602 time for calcul the mask position with numpy : 0.03293776512145996 nb_pixel_total : 23981 time to create 1 rle with old method : 0.02705669403076172 time for calcul the mask position with numpy : 0.02953362464904785 nb_pixel_total : 33113 time to create 1 rle with old method : 0.03644728660583496 time for calcul the mask position with numpy : 0.02768564224243164 nb_pixel_total : 5498 time to create 1 rle with old method : 0.006132841110229492 create new chi : 4.251561403274536 time to delete rle : 0.0020673274993896484 batch 1 Loaded 51 chid ids of type : 3594 ++++++++++++++++++++++++++++++Number RLEs to save : 13338 TO DO : save crop sub photo not yet done ! save time : 1.4958136081695557 map_output_result : {1334600578: (0.0, 'Should be the crop_list due to order', 0), 1334600573: (0.0, 'Should be the crop_list due to order', 0), 1334600570: (0.0, 'Should be the crop_list due to order', 0), 1334600563: (0.0, 'Should be the crop_list due to order', 0), 1334600552: (0.0, 'Should be the crop_list due to order', 0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1334600578, 1334600573, 1334600570, 1334600563, 1334600552] Looping around the photos to save general results len do output : 5 /1334600578.Didn't retrieve data . /1334600573.Didn't retrieve data . /1334600570.Didn't retrieve data . /1334600563.Didn't retrieve data . /1334600552.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600578', None, None, None, None, None, '2636519') ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600573', None, None, None, None, None, '2636519') ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600570', None, None, None, None, None, '2636519') ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600563', None, None, None, None, None, '2636519') ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600552', None, None, None, None, None, '2636519') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 15 time used for this insertion : 0.01277470588684082 save_final save missing photos in datou_result : time spend for datou_step_exec : 93.29356217384338 time spend to save output : 0.013233423233032227 total time spend for step 3 : 93.30679559707642 step4:ventilate_hashtags_in_portfolio Wed Mar 5 10:04:42 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! Iterating over portfolio : 21053304 get user id for portfolio 21053304 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21053304 AND mptpi.`type`=3594 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('carton','pehd','environnement','papier','pet_fonce','autre','background','metal','pet_clair','mal_croppe','flou')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21053304 AND mptpi.`type`=3594 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('carton','pehd','environnement','papier','pet_fonce','autre','background','metal','pet_clair','mal_croppe','flou')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21053304 AND mptpi.`type`=3594 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('carton','pehd','environnement','papier','pet_fonce','autre','background','metal','pet_clair','mal_croppe','flou')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/21055996,21055997,21055998,21055999,21056000,21056001,21056002,21056003,21056004,21056005,21056006?tags=carton,pehd,environnement,papier,pet_fonce,autre,background,metal,pet_clair,mal_croppe,flou Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1334600578, 1334600573, 1334600570, 1334600563, 1334600552] Looping around the photos to save general results len do output : 1 /21053304. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600578', None, None, None, None, None, '2636519') ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600573', None, None, None, None, None, '2636519') ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600570', None, None, None, None, None, '2636519') ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600563', None, None, None, None, None, '2636519') ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600552', None, None, None, None, None, '2636519') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 6 time used for this insertion : 0.21318602561950684 save_final save missing photos in datou_result : time spend for datou_step_exec : 9.520052671432495 time spend to save output : 0.2134397029876709 total time spend for step 4 : 9.733492374420166 step5:final Wed Mar 5 10:04:51 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1334600578: ('0.08810448013117284',), 1334600573: ('0.08810448013117284',), 1334600570: ('0.08810448013117284',), 1334600563: ('0.08810448013117284',), 1334600552: ('0.08810448013117284',)} new output for save of step final : {1334600578: ('0.08810448013117284',), 1334600573: ('0.08810448013117284',), 1334600570: ('0.08810448013117284',), 1334600563: ('0.08810448013117284',), 1334600552: ('0.08810448013117284',)} [1334600578, 1334600573, 1334600570, 1334600563, 1334600552] Looping around the photos to save general results len do output : 5 /1334600578.Didn't retrieve data . /1334600573.Didn't retrieve data . /1334600570.Didn't retrieve data . /1334600563.Didn't retrieve data . /1334600552.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600578', None, None, None, None, None, '2636519') ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600573', None, None, None, None, None, '2636519') ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600570', None, None, None, None, None, '2636519') ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600563', None, None, None, None, None, '2636519') ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600552', None, None, None, None, None, '2636519') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 15 time used for this insertion : 0.012949705123901367 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.2183222770690918 time spend to save output : 0.013355255126953125 total time spend for step 5 : 1.231677532196045 step6:blur_detection Wed Mar 5 10:04:53 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure inside step blur_detection toutes les photos sont déjà traitées, on saute les calculs Inside saveOutput : final : False verbose : 0 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 5 time used for this insertion : 0.009610652923583984 begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 5 time used for this insertion : 0.008928537368774414 save missing photos in datou_result : time spend for datou_step_exec : 0.29026103019714355 time spend to save output : 0.02276778221130371 total time spend for step 6 : 0.31302881240844727 step7:brightness Wed Mar 5 10:04:53 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure inside step calcul brightness toutes les photos sont déjà traitées, on saute les calculs Inside saveOutput : final : False verbose : 0 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 5 time used for this insertion : 0.00792551040649414 begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 5 time used for this insertion : 0.007846593856811523 save missing photos in datou_result : time spend for datou_step_exec : 0.08110332489013672 time spend to save output : 0.02049541473388672 total time spend for step 7 : 0.10159873962402344 step8:velours_tree Wed Mar 5 10:04:53 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.16232728958129883 time spend to save output : 7.462501525878906e-05 total time spend for step 8 : 0.16240191459655762 step9:send_mail_cod Wed Mar 5 10:04:53 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 complete output_args for input 1 complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P21053304_05-03-2025_10_04_53.pdf 21055996 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette210559961741165493 21055997 imagette210559971741165494 21055999 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette210559991741165494 21056000 change filename to text .imagette210560001741165496 21056001 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette210560011741165496 21056002 imagette210560021741165496 21056003 change filename to text .imagette210560031741165496 21056004 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette210560041741165496 21056005 imagette210560051741165497 21056006 imagette210560061741165497 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=21053304 and hashtag_type = 3594 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/21055996,21055997,21055998,21055999,21056000,21056001,21056002,21056003,21056004,21056005,21056006?tags=carton,pehd,environnement,papier,pet_fonce,autre,background,metal,pet_clair,mal_croppe,flou args[1334600578] : ((1334600578, -5.202334345933252, 492609224), (1334600578, -0.40630949780817405, 496442774), '0.08810448013117284') We are sending mail with results at report@fotonower.com args[1334600573] : ((1334600573, -6.135848526147168, 492609224), (1334600573, 0.06589760465785749, 2107752395), '0.08810448013117284') We are sending mail with results at report@fotonower.com args[1334600570] : ((1334600570, -6.200096639603137, 492609224), (1334600570, 0.11871423612181962, 2107752395), '0.08810448013117284') We are sending mail with results at report@fotonower.com args[1334600563] : ((1334600563, -6.087390636172522, 492609224), (1334600563, 0.12534476030130018, 2107752395), '0.08810448013117284') We are sending mail with results at report@fotonower.com args[1334600552] : ((1334600552, -6.284188849772036, 492609224), (1334600552, -0.04636138985297169, 2107752395), '0.08810448013117284') We are sending mail with results at report@fotonower.com refus_total : 0.08810448013117284 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=21053304 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21053304_05-03-2025_10_04_53.pdf results_Auto_P21053304_05-03-2025_10_04_53.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21053304_05-03-2025_10_04_53.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3318','21053304','results_Auto_P21053304_05-03-2025_10_04_53.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21053304_05-03-2025_10_04_53.pdf','pdf','','0.5','0.08810448013117284') message_in_mail: Bonjour,
Veuillez trouver ci dessous les résultats du service carac on demand pour le portfolio: https://www.fotonower.com/view/21053304

https://www.fotonower.com/image?json=false&list_photos_id=1334600578
Bravo, la photo est bien prise.
https://www.fotonower.com/image?json=false&list_photos_id=1334600573
Bravo, la photo est bien prise.
https://www.fotonower.com/image?json=false&list_photos_id=1334600570
Bravo, la photo est bien prise.
https://www.fotonower.com/image?json=false&list_photos_id=1334600563
Bravo, la photo est bien prise.
https://www.fotonower.com/image?json=false&list_photos_id=1334600552
Bravo, la photo est bien prise.

Dans ces conditions,le taux de refus est: 8.81%
Veuillez trouver les photos des contaminants.

exemples de contaminants: carton: https://www.fotonower.com/view/21055996?limit=200
exemples de contaminants: papier: https://www.fotonower.com/view/21055999?limit=200
exemples de contaminants: pet_fonce: https://www.fotonower.com/view/21056000?limit=200
exemples de contaminants: autre: https://www.fotonower.com/view/21056001?limit=200
exemples de contaminants: metal: https://www.fotonower.com/view/21056003?limit=200
exemples de contaminants: pet_clair: https://www.fotonower.com/view/21056004?limit=200
Veuillez trouver le rapport en pdf:https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21053304_05-03-2025_10_04_53.pdf.

Lien vers velours :https://www.fotonower.com/velours/21055996,21055997,21055998,21055999,21056000,21056001,21056002,21056003,21056004,21056005,21056006?tags=carton,pehd,environnement,papier,pet_fonce,autre,background,metal,pet_clair,mal_croppe,flou.


L'équipe Fotonower 202 b'' Server: nginx Date: Wed, 05 Mar 2025 09:04:59 GMT Content-Length: 0 Connection: close X-Message-Id: F2_MI6XuRm6kyVqd3mVQpg Access-Control-Allow-Origin: https://sendgrid.api-docs.io Access-Control-Allow-Methods: POST Access-Control-Allow-Headers: Authorization, Content-Type, On-behalf-of, x-sg-elas-acl Access-Control-Max-Age: 600 X-No-CORS-Reason: https://sendgrid.com/docs/Classroom/Basics/API/cors.html Strict-Transport-Security: max-age=31536000; includeSubDomains Content-Security-Policy: frame-ancestors 'none' Cache-Control: no-cache X-Content-Type-Options: no-sniff Referrer-Policy: strict-origin-when-cross-origin Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1334600578, 1334600573, 1334600570, 1334600563, 1334600552] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600578', None, None, None, None, None, '2636519') ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600573', None, None, None, None, None, '2636519') ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600570', None, None, None, None, None, '2636519') ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600563', None, None, None, None, None, '2636519') ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600552', None, None, None, None, None, '2636519') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 5 time used for this insertion : 0.012346267700195312 save_final save missing photos in datou_result : time spend for datou_step_exec : 6.123280763626099 time spend to save output : 0.012688398361206055 total time spend for step 9 : 6.135969161987305 step10:split_time_score Wed Mar 5 10:04:59 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('17', 5),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 04022025 21053304 Nombre de photos uploadées : 5 / 23040 (0%) 04022025 21053304 Nombre de photos taguées (types de déchets): 0 / 5 (0%) 04022025 21053304 Nombre de photos taguées (volume) : 0 / 5 (0%) elapsed_time : load_data_split_time_score 2.384185791015625e-06 elapsed_time : order_list_meta_photo_and_scores 6.198883056640625e-06 ????? elapsed_time : fill_and_build_computed_from_old_data 0.0003380775451660156 elapsed_time : insert_dashboard_record_day_entry 0.02358269691467285 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21053265 order by id desc limit 1 Qualite : 0.023622352448004997 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21053273_05-03-2025_08_56_03.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21053273 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21053273 AND mptpi.`type`=3726 To do Qualite : 0.03717048175398119 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21053278_05-03-2025_08_54_26.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21053278 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21053278 AND mptpi.`type`=3726 To do Qualite : 0.03664463714671318 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21053291_05-03-2025_08_58_17.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21053291 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21053291 AND mptpi.`type`=3726 To do Qualite : 0.08810448013117284 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21053304_05-03-2025_10_04_53.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21053304 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21053304 AND mptpi.`type`=3594 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'04022025': {'nb_upload': 5, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1334600578, 1334600573, 1334600570, 1334600563, 1334600552] Looping around the photos to save general results len do output : 1 /21053304Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600578', None, None, None, None, None, '2636519') ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600573', None, None, None, None, None, '2636519') ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600570', None, None, None, None, None, '2636519') ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600563', None, None, None, None, None, '2636519') ('3318', None, None, None, None, None, None, None, '2636519') ('3318', '21053304', '1334600552', None, None, None, None, None, '2636519') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 6 time used for this insertion : 0.012101888656616211 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.2243962287902832 time spend to save output : 0.012466669082641602 total time spend for step 10 : 1.2368628978729248 caffe_path_current : About to save ! 2 After save, about to update current ! ret : 2 len(input) + len(total_photo_id_missing) : 5 set_done_treatment 80.43user 108.95system 4:34.90elapsed 68%CPU (0avgtext+0avgdata 4385440maxresident)k 891152inputs+71880outputs (12097major+12477712minor)pagefaults 0swaps