python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2043392 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 30>10 please execute split_portfolio.py -i 20973396 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2629128'] with mtr_portfolio_ids : ['20973396'] and first list_photo_ids : [] new path : /proc/2043392/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 1.3607070446014404 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Mon Mar 3 12:12:35 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 2964 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-03-03 12:12:39.958546: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-03 12:12:39.991235: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-03 12:12:39.992875: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fc124000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-03 12:12:39.992920: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-03 12:12:39.997093: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-03 12:12:40.255071: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5b2a080 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-03 12:12:40.255110: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-03 12:12:40.256222: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-03 12:12:40.256497: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-03 12:12:40.258353: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-03 12:12:40.275361: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-03 12:12:40.275702: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-03 12:12:40.308000: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-03 12:12:40.313204: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-03 12:12:40.369039: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-03 12:12:40.370206: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-03 12:12:40.370615: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-03 12:12:40.371253: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-03 12:12:40.371274: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-03 12:12:40.371284: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-03 12:12:40.372698: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2512 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-03-03 12:12:40.680019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-03 12:12:40.680106: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-03 12:12:40.680127: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-03 12:12:40.680146: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-03 12:12:40.680164: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-03 12:12:40.680182: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-03 12:12:40.680199: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-03 12:12:40.680217: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-03 12:12:40.681190: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-03 12:12:40.682174: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-03 12:12:40.682207: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-03 12:12:40.682226: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-03 12:12:40.682243: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-03 12:12:40.682260: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-03 12:12:40.682277: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-03 12:12:40.682294: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-03 12:12:40.682311: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-03 12:12:40.683303: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-03 12:12:40.683336: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-03 12:12:40.683346: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-03 12:12:40.683356: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-03 12:12:40.684293: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2512 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-03-03 12:12:50.252895: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-03 12:12:50.446095: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-03 12:12:52.233278: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-03 12:12:52.233329: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-03 12:12:52.239732: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-03 12:12:52.239755: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-03 12:12:52.288317: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-03 12:12:52.288364: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-03 12:12:52.330172: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.09GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-03 12:12:52.330197: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.09GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-03 12:12:52.381205: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.15GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-03 12:12:52.381233: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.15GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-03 12:12:52.383503: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.45G (1562247168 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.701959: W tensorflow/core/common_runtime/bfc_allocator.cc:311] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature. 2025-03-03 12:12:52.750381: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.751683: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.752837: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.753655: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.777033: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.778206: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.779443: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.780626: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.781813: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.783014: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.797880: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.798518: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.833134: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.833256: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-03-03 12:12:52.834476: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.835633: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.842990: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.843643: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.851976: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.852658: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.871772: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.872614: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.873365: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.874080: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.878691: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.879477: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.880221: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.880949: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.882561: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.892919: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.893669: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.903392: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.904187: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.904899: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.905618: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.906391: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-03 12:12:52.907130: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.95G (2099118080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 9 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 12 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 14 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 12 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 10 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 9 Detection mask done ! Trying to reset tf kernel 2044248 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 1992 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 3185 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.012588739395141602 nb_pixel_total : 574792 time to create 1 rle with new method : 0.06597661972045898 length of segment : 1953 time for calcul the mask position with numpy : 0.0015571117401123047 nb_pixel_total : 137546 time to create 1 rle with old method : 0.155989408493042 length of segment : 391 time for calcul the mask position with numpy : 0.00389862060546875 nb_pixel_total : 282921 time to create 1 rle with new method : 0.010137319564819336 length of segment : 929 time for calcul the mask position with numpy : 0.004824399948120117 nb_pixel_total : 326022 time to create 1 rle with new method : 0.010185718536376953 length of segment : 1184 time for calcul the mask position with numpy : 0.0008893013000488281 nb_pixel_total : 49323 time to create 1 rle with old method : 0.058516502380371094 length of segment : 392 time for calcul the mask position with numpy : 0.009164094924926758 nb_pixel_total : 486776 time to create 1 rle with new method : 0.024706602096557617 length of segment : 1557 time for calcul the mask position with numpy : 0.0008845329284667969 nb_pixel_total : 34355 time to create 1 rle with old method : 0.03985953330993652 length of segment : 295 time for calcul the mask position with numpy : 0.0010097026824951172 nb_pixel_total : 66977 time to create 1 rle with old method : 0.0782628059387207 length of segment : 448 time for calcul the mask position with numpy : 0.0017230510711669922 nb_pixel_total : 99022 time to create 1 rle with old method : 0.11775755882263184 length of segment : 339 time for calcul the mask position with numpy : 0.0002048015594482422 nb_pixel_total : 10503 time to create 1 rle with old method : 0.012888193130493164 length of segment : 110 time for calcul the mask position with numpy : 0.00018334388732910156 nb_pixel_total : 7017 time to create 1 rle with old method : 0.009205102920532227 length of segment : 80 time for calcul the mask position with numpy : 0.0002334117889404297 nb_pixel_total : 14138 time to create 1 rle with old method : 0.01619410514831543 length of segment : 185 time for calcul the mask position with numpy : 0.0010907649993896484 nb_pixel_total : 35377 time to create 1 rle with old method : 0.04088926315307617 length of segment : 180 time for calcul the mask position with numpy : 0.005323171615600586 nb_pixel_total : 224547 time to create 1 rle with new method : 0.012420654296875 length of segment : 881 time for calcul the mask position with numpy : 0.013010978698730469 nb_pixel_total : 661066 time to create 1 rle with new method : 0.032538652420043945 length of segment : 1841 time for calcul the mask position with numpy : 0.0010476112365722656 nb_pixel_total : 34845 time to create 1 rle with old method : 0.0480043888092041 length of segment : 225 time for calcul the mask position with numpy : 0.00048613548278808594 nb_pixel_total : 16797 time to create 1 rle with old method : 0.020635366439819336 length of segment : 361 time for calcul the mask position with numpy : 0.0007889270782470703 nb_pixel_total : 27669 time to create 1 rle with old method : 0.036347389221191406 length of segment : 272 time for calcul the mask position with numpy : 0.004061460494995117 nb_pixel_total : 326788 time to create 1 rle with new method : 0.011233091354370117 length of segment : 1082 time for calcul the mask position with numpy : 0.012892961502075195 nb_pixel_total : 306923 time to create 1 rle with new method : 0.025263547897338867 length of segment : 719 time for calcul the mask position with numpy : 0.0021982192993164062 nb_pixel_total : 171140 time to create 1 rle with new method : 0.005415678024291992 length of segment : 406 time for calcul the mask position with numpy : 0.0010349750518798828 nb_pixel_total : 56201 time to create 1 rle with old method : 0.06656932830810547 length of segment : 451 time for calcul the mask position with numpy : 0.001233816146850586 nb_pixel_total : 79831 time to create 1 rle with old method : 0.09617757797241211 length of segment : 551 time for calcul the mask position with numpy : 0.0015993118286132812 nb_pixel_total : 131517 time to create 1 rle with old method : 0.15756916999816895 length of segment : 373 time for calcul the mask position with numpy : 8.153915405273438e-05 nb_pixel_total : 2282 time to create 1 rle with old method : 0.002844572067260742 length of segment : 86 time for calcul the mask position with numpy : 0.0014998912811279297 nb_pixel_total : 114481 time to create 1 rle with old method : 0.14318370819091797 length of segment : 401 time for calcul the mask position with numpy : 0.003620147705078125 nb_pixel_total : 296795 time to create 1 rle with new method : 0.008804559707641602 length of segment : 964 time for calcul the mask position with numpy : 0.0026602745056152344 nb_pixel_total : 99507 time to create 1 rle with old method : 0.1199338436126709 length of segment : 423 time for calcul the mask position with numpy : 0.0004918575286865234 nb_pixel_total : 14153 time to create 1 rle with old method : 0.017579317092895508 length of segment : 160 time for calcul the mask position with numpy : 0.013342618942260742 nb_pixel_total : 459375 time to create 1 rle with new method : 0.025068998336791992 length of segment : 986 time for calcul the mask position with numpy : 0.0006113052368164062 nb_pixel_total : 30471 time to create 1 rle with old method : 0.037087440490722656 length of segment : 316 time for calcul the mask position with numpy : 0.000408172607421875 nb_pixel_total : 23641 time to create 1 rle with old method : 0.03089141845703125 length of segment : 124 time for calcul the mask position with numpy : 0.0004355907440185547 nb_pixel_total : 19844 time to create 1 rle with old method : 0.02394723892211914 length of segment : 194 time for calcul the mask position with numpy : 0.0014967918395996094 nb_pixel_total : 104058 time to create 1 rle with old method : 0.12112665176391602 length of segment : 686 time for calcul the mask position with numpy : 0.0016293525695800781 nb_pixel_total : 107397 time to create 1 rle with old method : 0.12979817390441895 length of segment : 392 time for calcul the mask position with numpy : 0.0010221004486083984 nb_pixel_total : 37751 time to create 1 rle with old method : 0.04591989517211914 length of segment : 340 time for calcul the mask position with numpy : 0.0005471706390380859 nb_pixel_total : 32784 time to create 1 rle with old method : 0.03966069221496582 length of segment : 320 time for calcul the mask position with numpy : 0.0005857944488525391 nb_pixel_total : 38085 time to create 1 rle with old method : 0.044010162353515625 length of segment : 331 time for calcul the mask position with numpy : 0.0006728172302246094 nb_pixel_total : 38372 time to create 1 rle with old method : 0.044020891189575195 length of segment : 313 time for calcul the mask position with numpy : 0.0011487007141113281 nb_pixel_total : 44534 time to create 1 rle with old method : 0.05005621910095215 length of segment : 421 time for calcul the mask position with numpy : 0.0013148784637451172 nb_pixel_total : 58885 time to create 1 rle with old method : 0.06520342826843262 length of segment : 428 time for calcul the mask position with numpy : 0.006599903106689453 nb_pixel_total : 436909 time to create 1 rle with new method : 0.011708259582519531 length of segment : 1213 time for calcul the mask position with numpy : 0.0005779266357421875 nb_pixel_total : 40958 time to create 1 rle with old method : 0.049178123474121094 length of segment : 237 time for calcul the mask position with numpy : 0.003522634506225586 nb_pixel_total : 307207 time to create 1 rle with new method : 0.007979869842529297 length of segment : 938 time for calcul the mask position with numpy : 0.00032067298889160156 nb_pixel_total : 12687 time to create 1 rle with old method : 0.01604485511779785 length of segment : 175 time for calcul the mask position with numpy : 0.0003647804260253906 nb_pixel_total : 15082 time to create 1 rle with old method : 0.01833319664001465 length of segment : 165 time for calcul the mask position with numpy : 0.00015997886657714844 nb_pixel_total : 2691 time to create 1 rle with old method : 0.0035343170166015625 length of segment : 42 time for calcul the mask position with numpy : 0.0003688335418701172 nb_pixel_total : 13429 time to create 1 rle with old method : 0.0177304744720459 length of segment : 124 time for calcul the mask position with numpy : 0.0003628730773925781 nb_pixel_total : 10315 time to create 1 rle with old method : 0.012187719345092773 length of segment : 94 time for calcul the mask position with numpy : 0.004041194915771484 nb_pixel_total : 165572 time to create 1 rle with new method : 0.00679326057434082 length of segment : 851 time for calcul the mask position with numpy : 0.0038733482360839844 nb_pixel_total : 138232 time to create 1 rle with old method : 0.1576385498046875 length of segment : 480 time for calcul the mask position with numpy : 0.009326457977294922 nb_pixel_total : 509075 time to create 1 rle with new method : 0.01794719696044922 length of segment : 1136 time for calcul the mask position with numpy : 0.0003027915954589844 nb_pixel_total : 7498 time to create 1 rle with old method : 0.009183406829833984 length of segment : 122 time for calcul the mask position with numpy : 0.00047206878662109375 nb_pixel_total : 18786 time to create 1 rle with old method : 0.022588014602661133 length of segment : 269 time for calcul the mask position with numpy : 0.00010466575622558594 nb_pixel_total : 4686 time to create 1 rle with old method : 0.005776643753051758 length of segment : 76 time for calcul the mask position with numpy : 0.0015425682067871094 nb_pixel_total : 50687 time to create 1 rle with old method : 0.057651519775390625 length of segment : 563 time for calcul the mask position with numpy : 0.0005099773406982422 nb_pixel_total : 22992 time to create 1 rle with old method : 0.027388811111450195 length of segment : 299 time for calcul the mask position with numpy : 9.918212890625e-05 nb_pixel_total : 2109 time to create 1 rle with old method : 0.0025272369384765625 length of segment : 128 time for calcul the mask position with numpy : 0.0001266002655029297 nb_pixel_total : 995 time to create 1 rle with old method : 0.0012950897216796875 length of segment : 50 time for calcul the mask position with numpy : 0.00033092498779296875 nb_pixel_total : 12502 time to create 1 rle with old method : 0.015346050262451172 length of segment : 98 time for calcul the mask position with numpy : 0.002351045608520508 nb_pixel_total : 125860 time to create 1 rle with old method : 0.14672136306762695 length of segment : 344 time for calcul the mask position with numpy : 0.0005819797515869141 nb_pixel_total : 15368 time to create 1 rle with old method : 0.02127814292907715 length of segment : 146 time for calcul the mask position with numpy : 0.0001342296600341797 nb_pixel_total : 3883 time to create 1 rle with old method : 0.004897594451904297 length of segment : 87 time for calcul the mask position with numpy : 0.016425371170043945 nb_pixel_total : 644803 time to create 1 rle with new method : 0.04507327079772949 length of segment : 1667 time for calcul the mask position with numpy : 0.00012946128845214844 nb_pixel_total : 3858 time to create 1 rle with old method : 0.004802703857421875 length of segment : 72 time for calcul the mask position with numpy : 0.004190683364868164 nb_pixel_total : 351194 time to create 1 rle with new method : 0.009207963943481445 length of segment : 1042 time for calcul the mask position with numpy : 0.00012350082397460938 nb_pixel_total : 4272 time to create 1 rle with old method : 0.0053768157958984375 length of segment : 73 time for calcul the mask position with numpy : 0.0005125999450683594 nb_pixel_total : 19474 time to create 1 rle with old method : 0.024076223373413086 length of segment : 172 time for calcul the mask position with numpy : 0.00015211105346679688 nb_pixel_total : 6594 time to create 1 rle with old method : 0.008185625076293945 length of segment : 95 time for calcul the mask position with numpy : 0.0002548694610595703 nb_pixel_total : 12962 time to create 1 rle with old method : 0.015902280807495117 length of segment : 187 time for calcul the mask position with numpy : 0.0006783008575439453 nb_pixel_total : 26838 time to create 1 rle with old method : 0.03245806694030762 length of segment : 191 time for calcul the mask position with numpy : 0.007898569107055664 nb_pixel_total : 343179 time to create 1 rle with new method : 0.014927864074707031 length of segment : 1760 time for calcul the mask position with numpy : 0.0005638599395751953 nb_pixel_total : 30226 time to create 1 rle with old method : 0.03512454032897949 length of segment : 171 time for calcul the mask position with numpy : 0.0004935264587402344 nb_pixel_total : 26269 time to create 1 rle with old method : 0.03185105323791504 length of segment : 137 time for calcul the mask position with numpy : 0.00043272972106933594 nb_pixel_total : 28196 time to create 1 rle with old method : 0.03430581092834473 length of segment : 142 time for calcul the mask position with numpy : 0.00018835067749023438 nb_pixel_total : 3825 time to create 1 rle with old method : 0.004842996597290039 length of segment : 85 time for calcul the mask position with numpy : 0.010664939880371094 nb_pixel_total : 587146 time to create 1 rle with new method : 0.0161285400390625 length of segment : 1505 time for calcul the mask position with numpy : 0.0013570785522460938 nb_pixel_total : 81476 time to create 1 rle with old method : 0.09716272354125977 length of segment : 324 time for calcul the mask position with numpy : 0.00020575523376464844 nb_pixel_total : 6016 time to create 1 rle with old method : 0.007422208786010742 length of segment : 217 time for calcul the mask position with numpy : 0.00017881393432617188 nb_pixel_total : 4715 time to create 1 rle with old method : 0.005972623825073242 length of segment : 73 time for calcul the mask position with numpy : 0.00033974647521972656 nb_pixel_total : 8182 time to create 1 rle with old method : 0.010499954223632812 length of segment : 162 time for calcul the mask position with numpy : 0.0005190372467041016 nb_pixel_total : 26161 time to create 1 rle with old method : 0.031014680862426758 length of segment : 235 time for calcul the mask position with numpy : 0.0008027553558349609 nb_pixel_total : 61341 time to create 1 rle with old method : 0.07200312614440918 length of segment : 212 time spent for convertir_results : 8.685826301574707 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 83 chid ids of type : 3663 Number RLEs to save : 38249 save missing photos in datou_result : time spend for datou_step_exec : 35.88603663444519 time spend to save output : 8.714020013809204 total time spend for step 1 : 44.600056648254395 step2:crop_condition Mon Mar 3 12:13:20 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 83 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 10 About to insert : list_path_to_insert length 10 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 17 About to insert : list_path_to_insert length 17 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1340612350, 1340612307, 1340612306, 1340612251, 1340612249, 1340612173, 1340612169, 1340612077, 1340612074, 1340612030] Looping around the photos to save general results len do output : 46 /-3692315542Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315547Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315566Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315576Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315594Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315593Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315600Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315601Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315605Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315619Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315556Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315585Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315613Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315621Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315572Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315586Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315615Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315622Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315623Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315552Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315549Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315550Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315551Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315563Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315564Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315568Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315577Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315579Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315596Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315597Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315588Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315589Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315610Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315608Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315611Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315618Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315553Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315574Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315595Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315587Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315607Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315603Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315620Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315569Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315583Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692315614Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612350', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612307', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612306', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612251', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612249', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612173', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612169', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612077', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612074', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612030', None, None, None, None, None, '2629128') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 148 time used for this insertion : 0.31472206115722656 save_final save missing photos in datou_result : time spend for datou_step_exec : 10.894607305526733 time spend to save output : 0.3173234462738037 total time spend for step 2 : 11.211930751800537 step3:thcl Mon Mar 3 12:13:31 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.005342960357666016 time to convert the images to numpy array : 0.006734132766723633 time to import caffe and check if the image exist : 0.008582592010498047 time to convert the images to numpy array : 0.02155137062072754 time to import caffe and check if the image exist : 0.008302450180053711 time to convert the images to numpy array : 0.02821040153503418 time to import caffe and check if the image exist : 0.007433891296386719 time to convert the images to numpy array : 0.029216766357421875 time to import caffe and check if the image exist : 0.010042905807495117 time to convert the images to numpy array : 0.031090974807739258 time to import caffe and check if the image exist : 0.011810302734375 time to convert the images to numpy array : 0.03105020523071289 time to import caffe and check if the image exist : 0.011332511901855469 time to convert the images to numpy array : 0.04065370559692383 time to import caffe and check if the image exist : 0.006500959396362305 time to convert the images to numpy array : 0.050984859466552734 time to import caffe and check if the image exist : 0.011461734771728516 time to convert the images to numpy array : 0.04601144790649414 time to import caffe and check if the image exist : 0.013413667678833008 time to convert the images to numpy array : 0.04448819160461426 total time to convert the images to numpy array : 0.24040937423706055 list photo_ids error: [] list photo_ids correct : [-3692315614, -3692315607, -3692315603, -3692315620, -3692315569, -3692315583, -3692315588, -3692315589, -3692315610, -3692315608, -3692315611, -3692315556, -3692315585, -3692315613, -3692315621, -3692315572, -3692315586, -3692315615, -3692315622, -3692315623, -3692315552, -3692315593, -3692315600, -3692315601, -3692315605, -3692315619, -3692315618, -3692315553, -3692315574, -3692315595, -3692315587, -3692315542, -3692315547, -3692315566, -3692315576, -3692315594, -3692315549, -3692315550, -3692315551, -3692315563, -3692315564, -3692315568, -3692315577, -3692315579, -3692315596, -3692315597] number of photos to traite : 46 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3185 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 2966 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.324007511138916 time used to do the prediction : 0.16210603713989258 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 2964 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 2964 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.3944966793060303 time used to do the prediction : 0.10820317268371582 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 1.239776611328125e-05 save missing photos in datou_result : time spend for datou_step_exec : 8.051255226135254 time spend to save output : 0.0003643035888671875 total time spend for step 3 : 8.051619529724121 step4:merge_mask_thcl_custom Mon Mar 3 12:13:39 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 83 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1340612350, 1340612307, 1340612306, 1340612251, 1340612249, 1340612173, 1340612169, 1340612077, 1340612074, 1340612030] Looping around the photos to save general results len do output : 10 /1340612350Didn't retrieve data .Didn't retrieve data . /1340612307Didn't retrieve data .Didn't retrieve data . /1340612306Didn't retrieve data .Didn't retrieve data . /1340612251Didn't retrieve data .Didn't retrieve data . /1340612249Didn't retrieve data .Didn't retrieve data . /1340612173Didn't retrieve data .Didn't retrieve data . /1340612169Didn't retrieve data .Didn't retrieve data . /1340612077Didn't retrieve data .Didn't retrieve data . /1340612074Didn't retrieve data .Didn't retrieve data . /1340612030Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612350', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612307', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612306', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612251', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612249', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612173', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612169', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612077', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612074', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612030', None, None, None, None, None, '2629128') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 6.046884536743164 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.09883236885070801 time spend to save output : 6.047429323196411 total time spend for step 4 : 6.146261692047119 step5:rle_unique_nms_with_priority Mon Mar 3 12:13:45 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 4 nb_hashtags : 2 time to prepare the origin masks : 1.40706467628479 time for calcul the mask position with numpy : 0.04099321365356445 nb_pixel_total : 962959 time to create 1 rle with new method : 0.10559940338134766 time for calcul the mask position with numpy : 0.012194633483886719 nb_pixel_total : 149012 time to create 1 rle with old method : 0.17586088180541992 time for calcul the mask position with numpy : 0.012717247009277344 nb_pixel_total : 235663 time to create 1 rle with new method : 0.4099864959716797 time for calcul the mask position with numpy : 0.013090133666992188 nb_pixel_total : 588656 time to create 1 rle with new method : 0.10482335090637207 time for calcul the mask position with numpy : 0.009503364562988281 nb_pixel_total : 137310 time to create 1 rle with old method : 0.15198111534118652 create new chi : 1.0609736442565918 time to delete rle : 0.013951778411865234 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 8224 TO DO : save crop sub photo not yet done ! save time : 0.602571964263916 nb_obj : 8 nb_hashtags : 5 time to prepare the origin masks : 2.6397526264190674 time for calcul the mask position with numpy : 0.04606008529663086 nb_pixel_total : 1261330 time to create 1 rle with new method : 0.4112701416015625 time for calcul the mask position with numpy : 0.0065534114837646484 nb_pixel_total : 10447 time to create 1 rle with old method : 0.01185917854309082 time for calcul the mask position with numpy : 0.00609135627746582 nb_pixel_total : 6969 time to create 1 rle with old method : 0.00818777084350586 time for calcul the mask position with numpy : 0.006470680236816406 nb_pixel_total : 34144 time to create 1 rle with old method : 0.03846859931945801 time for calcul the mask position with numpy : 0.006306171417236328 nb_pixel_total : 4027 time to create 1 rle with old method : 0.0047838687896728516 time for calcul the mask position with numpy : 0.0066568851470947266 nb_pixel_total : 98798 time to create 1 rle with old method : 0.12530851364135742 time for calcul the mask position with numpy : 0.007264852523803711 nb_pixel_total : 66855 time to create 1 rle with old method : 0.08400964736938477 time for calcul the mask position with numpy : 0.010474443435668945 nb_pixel_total : 542168 time to create 1 rle with new method : 0.23715925216674805 time for calcul the mask position with numpy : 0.007157087326049805 nb_pixel_total : 48862 time to create 1 rle with old method : 0.0666358470916748 create new chi : 1.1002416610717773 time to delete rle : 0.0006706714630126953 batch 1 Loaded 9 chid ids of type : 3726 Number RLEs to save : 6396 TO DO : save crop sub photo not yet done ! save time : 0.4090108871459961 nb_obj : 7 nb_hashtags : 3 time to prepare the origin masks : 0.6678528785705566 time for calcul the mask position with numpy : 0.013819217681884766 nb_pixel_total : 1009385 time to create 1 rle with new method : 0.03282618522644043 time for calcul the mask position with numpy : 0.006507396697998047 nb_pixel_total : 35230 time to create 1 rle with old method : 0.03987765312194824 time for calcul the mask position with numpy : 0.0067539215087890625 nb_pixel_total : 129270 time to create 1 rle with old method : 0.15963506698608398 time for calcul the mask position with numpy : 0.007542848587036133 nb_pixel_total : 9167 time to create 1 rle with old method : 0.013906478881835938 time for calcul the mask position with numpy : 0.007481575012207031 nb_pixel_total : 16718 time to create 1 rle with old method : 0.021936655044555664 time for calcul the mask position with numpy : 0.010530233383178711 nb_pixel_total : 614611 time to create 1 rle with new method : 0.05590009689331055 time for calcul the mask position with numpy : 0.008253812789916992 nb_pixel_total : 224473 time to create 1 rle with new method : 0.027944564819335938 time for calcul the mask position with numpy : 0.006096363067626953 nb_pixel_total : 34746 time to create 1 rle with old method : 0.03868412971496582 create new chi : 0.46561551094055176 time to delete rle : 0.0007319450378417969 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 8632 TO DO : save crop sub photo not yet done ! save time : 5.844936847686768 nb_obj : 9 nb_hashtags : 3 time to prepare the origin masks : 1.1967620849609375 time for calcul the mask position with numpy : 0.040418148040771484 nb_pixel_total : 1139074 time to create 1 rle with new method : 0.5195038318634033 time for calcul the mask position with numpy : 0.006398439407348633 nb_pixel_total : 73112 time to create 1 rle with old method : 0.07937955856323242 time for calcul the mask position with numpy : 0.005895853042602539 nb_pixel_total : 17508 time to create 1 rle with old method : 0.019366741180419922 time for calcul the mask position with numpy : 0.006841421127319336 nb_pixel_total : 250968 time to create 1 rle with new method : 0.08322715759277344 time for calcul the mask position with numpy : 0.0062618255615234375 nb_pixel_total : 99292 time to create 1 rle with old method : 0.1086118221282959 time for calcul the mask position with numpy : 0.006309986114501953 nb_pixel_total : 114332 time to create 1 rle with old method : 0.1276702880859375 time for calcul the mask position with numpy : 0.009990453720092773 nb_pixel_total : 2273 time to create 1 rle with old method : 0.002602815628051758 time for calcul the mask position with numpy : 0.010160207748413086 nb_pixel_total : 56101 time to create 1 rle with old method : 0.06078815460205078 time for calcul the mask position with numpy : 0.009496688842773438 nb_pixel_total : 11979 time to create 1 rle with old method : 0.013184547424316406 time for calcul the mask position with numpy : 0.011287450790405273 nb_pixel_total : 308961 time to create 1 rle with new method : 0.03836488723754883 create new chi : 1.1751091480255127 time to delete rle : 0.0007486343383789062 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 8447 TO DO : save crop sub photo not yet done ! save time : 3.902982711791992 nb_obj : 11 nb_hashtags : 3 time to prepare the origin masks : 0.9295535087585449 time for calcul the mask position with numpy : 0.021924734115600586 nb_pixel_total : 1332006 time to create 1 rle with new method : 0.10092020034790039 time for calcul the mask position with numpy : 0.006546735763549805 nb_pixel_total : 1795 time to create 1 rle with old method : 0.0024573802947998047 time for calcul the mask position with numpy : 0.00639033317565918 nb_pixel_total : 14146 time to create 1 rle with old method : 0.016152381896972656 time for calcul the mask position with numpy : 0.006713390350341797 nb_pixel_total : 23541 time to create 1 rle with old method : 0.02681899070739746 time for calcul the mask position with numpy : 0.006984710693359375 nb_pixel_total : 13878 time to create 1 rle with old method : 0.015819072723388672 time for calcul the mask position with numpy : 0.006627559661865234 nb_pixel_total : 105828 time to create 1 rle with old method : 0.11950802803039551 time for calcul the mask position with numpy : 0.0062618255615234375 nb_pixel_total : 32507 time to create 1 rle with old method : 0.03825521469116211 time for calcul the mask position with numpy : 0.006514310836791992 nb_pixel_total : 19803 time to create 1 rle with old method : 0.0250856876373291 time for calcul the mask position with numpy : 0.007743120193481445 nb_pixel_total : 30413 time to create 1 rle with old method : 0.035655975341796875 time for calcul the mask position with numpy : 0.006160736083984375 nb_pixel_total : 3545 time to create 1 rle with old method : 0.004324436187744141 time for calcul the mask position with numpy : 0.008891820907592773 nb_pixel_total : 458423 time to create 1 rle with new method : 0.2643096446990967 time for calcul the mask position with numpy : 0.006390571594238281 nb_pixel_total : 37715 time to create 1 rle with old method : 0.04259920120239258 create new chi : 0.8003029823303223 time to delete rle : 0.0007054805755615234 batch 1 Loaded 12 chid ids of type : 3726 Number RLEs to save : 7602 TO DO : save crop sub photo not yet done ! save time : 0.5561258792877197 nb_obj : 7 nb_hashtags : 3 time to prepare the origin masks : 0.4295539855957031 time for calcul the mask position with numpy : 0.014328718185424805 nb_pixel_total : 1163395 time to create 1 rle with new method : 0.03216195106506348 time for calcul the mask position with numpy : 0.006206512451171875 nb_pixel_total : 11381 time to create 1 rle with old method : 0.012989044189453125 time for calcul the mask position with numpy : 0.007456541061401367 nb_pixel_total : 305549 time to create 1 rle with new method : 0.03019571304321289 time for calcul the mask position with numpy : 0.0060994625091552734 nb_pixel_total : 40817 time to create 1 rle with old method : 0.046889543533325195 time for calcul the mask position with numpy : 0.006455421447753906 nb_pixel_total : 15031 time to create 1 rle with old method : 0.01706528663635254 time for calcul the mask position with numpy : 0.00879669189453125 nb_pixel_total : 434188 time to create 1 rle with new method : 0.029671907424926758 time for calcul the mask position with numpy : 0.0061800479888916016 nb_pixel_total : 58760 time to create 1 rle with old method : 0.0646049976348877 time for calcul the mask position with numpy : 0.00616765022277832 nb_pixel_total : 44479 time to create 1 rle with old method : 0.05131864547729492 create new chi : 0.3548564910888672 time to delete rle : 0.0007250308990478516 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 7990 TO DO : save crop sub photo not yet done ! save time : 0.6129696369171143 nb_obj : 12 nb_hashtags : 5 time to prepare the origin masks : 2.749462604522705 time for calcul the mask position with numpy : 0.04763364791870117 nb_pixel_total : 1268834 time to create 1 rle with new method : 0.650496244430542 time for calcul the mask position with numpy : 0.006587982177734375 nb_pixel_total : 13352 time to create 1 rle with old method : 0.015475034713745117 time for calcul the mask position with numpy : 0.0062944889068603516 nb_pixel_total : 7485 time to create 1 rle with old method : 0.008505105972290039 time for calcul the mask position with numpy : 0.00602412223815918 nb_pixel_total : 2102 time to create 1 rle with old method : 0.002404928207397461 time for calcul the mask position with numpy : 0.006378889083862305 nb_pixel_total : 42308 time to create 1 rle with old method : 0.04809284210205078 time for calcul the mask position with numpy : 0.0059778690338134766 nb_pixel_total : 4658 time to create 1 rle with old method : 0.005711555480957031 time for calcul the mask position with numpy : 0.006190776824951172 nb_pixel_total : 4396 time to create 1 rle with old method : 0.005347728729248047 time for calcul the mask position with numpy : 0.008211374282836914 nb_pixel_total : 395272 time to create 1 rle with new method : 0.08668160438537598 time for calcul the mask position with numpy : 0.006434202194213867 nb_pixel_total : 18778 time to create 1 rle with old method : 0.02185368537902832 time for calcul the mask position with numpy : 0.0075225830078125 nb_pixel_total : 137909 time to create 1 rle with old method : 0.15855765342712402 time for calcul the mask position with numpy : 0.008877992630004883 nb_pixel_total : 165543 time to create 1 rle with new method : 0.03216385841369629 time for calcul the mask position with numpy : 0.007375240325927734 nb_pixel_total : 10282 time to create 1 rle with old method : 0.01698780059814453 time for calcul the mask position with numpy : 0.0070459842681884766 nb_pixel_total : 2681 time to create 1 rle with old method : 0.004535675048828125 create new chi : 1.1987862586975098 time to delete rle : 0.0012617111206054688 batch 1 Loaded 13 chid ids of type : 3726 Number RLEs to save : 8459 TO DO : save crop sub photo not yet done ! save time : 4.703248977661133 nb_obj : 11 nb_hashtags : 5 time to prepare the origin masks : 0.7989428043365479 time for calcul the mask position with numpy : 0.013969659805297852 nb_pixel_total : 1123256 time to create 1 rle with new method : 0.059099435806274414 time for calcul the mask position with numpy : 0.0060291290283203125 nb_pixel_total : 12928 time to create 1 rle with old method : 0.01484990119934082 time for calcul the mask position with numpy : 0.006059408187866211 nb_pixel_total : 10030 time to create 1 rle with old method : 0.01135396957397461 time for calcul the mask position with numpy : 0.00636744499206543 nb_pixel_total : 19300 time to create 1 rle with old method : 0.02281808853149414 time for calcul the mask position with numpy : 0.006459236145019531 nb_pixel_total : 1630 time to create 1 rle with old method : 0.0023658275604248047 time for calcul the mask position with numpy : 0.0064623355865478516 nb_pixel_total : 44174 time to create 1 rle with old method : 0.054903268814086914 time for calcul the mask position with numpy : 0.011161327362060547 nb_pixel_total : 713024 time to create 1 rle with new method : 0.18445277214050293 time for calcul the mask position with numpy : 0.005926370620727539 nb_pixel_total : 257 time to create 1 rle with old method : 0.00033354759216308594 time for calcul the mask position with numpy : 0.006692171096801758 nb_pixel_total : 125585 time to create 1 rle with old method : 0.1461026668548584 time for calcul the mask position with numpy : 0.006419181823730469 nb_pixel_total : 4252 time to create 1 rle with old method : 0.005172252655029297 time for calcul the mask position with numpy : 0.006258964538574219 nb_pixel_total : 15295 time to create 1 rle with old method : 0.017615795135498047 time for calcul the mask position with numpy : 0.006296396255493164 nb_pixel_total : 3869 time to create 1 rle with old method : 0.004448413848876953 create new chi : 0.6197984218597412 time to delete rle : 0.0006220340728759766 batch 1 Loaded 12 chid ids of type : 3726 Number RLEs to save : 6668 TO DO : save crop sub photo not yet done ! save time : 0.4994041919708252 nb_obj : 5 nb_hashtags : 4 time to prepare the origin masks : 0.5193479061126709 time for calcul the mask position with numpy : 0.016780376434326172 nb_pixel_total : 1643390 time to create 1 rle with new method : 0.03035879135131836 time for calcul the mask position with numpy : 0.006037473678588867 nb_pixel_total : 26807 time to create 1 rle with old method : 0.03002476692199707 time for calcul the mask position with numpy : 0.0060961246490478516 nb_pixel_total : 32561 time to create 1 rle with old method : 0.03508734703063965 time for calcul the mask position with numpy : 0.005907535552978516 nb_pixel_total : 2005 time to create 1 rle with old method : 0.002410888671875 time for calcul the mask position with numpy : 0.006123065948486328 nb_pixel_total : 26198 time to create 1 rle with old method : 0.029170751571655273 time for calcul the mask position with numpy : 0.008428096771240234 nb_pixel_total : 342639 time to create 1 rle with new method : 0.02830338478088379 create new chi : 0.20550251007080078 time to delete rle : 0.0005128383636474609 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 5673 TO DO : save crop sub photo not yet done ! save time : 0.4793729782104492 nb_obj : 7 nb_hashtags : 6 time to prepare the origin masks : 0.4926145076751709 time for calcul the mask position with numpy : 0.025959253311157227 nb_pixel_total : 1357087 time to create 1 rle with new method : 0.08875465393066406 time for calcul the mask position with numpy : 0.006196022033691406 nb_pixel_total : 5324 time to create 1 rle with old method : 0.006844520568847656 time for calcul the mask position with numpy : 0.006053924560546875 nb_pixel_total : 8121 time to create 1 rle with old method : 0.009246587753295898 time for calcul the mask position with numpy : 0.006590366363525391 nb_pixel_total : 81062 time to create 1 rle with old method : 0.09108352661132812 time for calcul the mask position with numpy : 0.005934000015258789 nb_pixel_total : 6007 time to create 1 rle with old method : 0.006677150726318359 time for calcul the mask position with numpy : 0.0060272216796875 nb_pixel_total : 24724 time to create 1 rle with old method : 0.02753901481628418 time for calcul the mask position with numpy : 0.006025075912475586 nb_pixel_total : 4707 time to create 1 rle with old method : 0.005373954772949219 time for calcul the mask position with numpy : 0.009658575057983398 nb_pixel_total : 586568 time to create 1 rle with new method : 0.23683524131774902 create new chi : 0.5532510280609131 time to delete rle : 0.0006992816925048828 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 6333 TO DO : save crop sub photo not yet done ! save time : 1.9715065956115723 map_output_result : {1340612350: (0.0, 'Should be the crop_list due to order', 0), 1340612307: (0.0, 'Should be the crop_list due to order', 0), 1340612306: (0.0, 'Should be the crop_list due to order', 0), 1340612251: (0.0, 'Should be the crop_list due to order', 0), 1340612249: (0.0, 'Should be the crop_list due to order', 0), 1340612173: (0.0, 'Should be the crop_list due to order', 0), 1340612169: (0.0, 'Should be the crop_list due to order', 0), 1340612077: (0.0, 'Should be the crop_list due to order', 0), 1340612074: (0.0, 'Should be the crop_list due to order', 0), 1340612030: (0.0, 'Should be the crop_list due to order', 0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1340612350, 1340612307, 1340612306, 1340612251, 1340612249, 1340612173, 1340612169, 1340612077, 1340612074, 1340612030] Looping around the photos to save general results len do output : 10 /1340612350.Didn't retrieve data . /1340612307.Didn't retrieve data . /1340612306.Didn't retrieve data . /1340612251.Didn't retrieve data . /1340612249.Didn't retrieve data . /1340612173.Didn't retrieve data . /1340612169.Didn't retrieve data . /1340612077.Didn't retrieve data . /1340612074.Didn't retrieve data . /1340612030.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612350', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612307', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612306', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612251', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612249', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612173', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612169', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612077', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612074', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612030', None, None, None, None, None, '2629128') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013481378555297852 save_final save missing photos in datou_result : time spend for datou_step_exec : 39.31814408302307 time spend to save output : 0.014014005661010742 total time spend for step 5 : 39.33215808868408 step6:crop_condition Mon Mar 3 12:14:25 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 91 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741000468_2043392 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.4484248161315918 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741000470_2043392 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.9644966125488281 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741000472_2043392 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 5.321675539016724 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! About to upload 8 photos upload in portfolio : 4869462 init cache_photo without model_param we have 8 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741000479_2043392 we have uploaded 8 photos in the portfolio 4869462 time of upload the photos Elapsed time : 7.104076862335205 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741000487_2043392 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.22822642326355 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741000490_2043392 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.0659730434417725 we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1340612350, 1340612307, 1340612306, 1340612251, 1340612249, 1340612173, 1340612169, 1340612077, 1340612074, 1340612030] Looping around the photos to save general results len do output : 21 /1340615800Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340615801Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340615802Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340615803Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340615805Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340615807Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340615808Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340615809Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340615810Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340615811Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340615815Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340615816Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340615817Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340615821Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340615822Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340615823Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340615824Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340615825Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340615828Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340615829Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340615831Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612350', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612307', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612306', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612251', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612249', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612173', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612169', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612077', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612074', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612030', None, None, None, None, None, '2629128') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 73 time used for this insertion : 0.8363132476806641 save_final save missing photos in datou_result : time spend for datou_step_exec : 27.036094188690186 time spend to save output : 0.8371000289916992 total time spend for step 6 : 27.873194217681885 step7:ventilate_hashtags_in_portfolio Mon Mar 3 12:14:53 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20973396 get user id for portfolio 20973396 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20973396 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Teint_Dans_La_Masse','metal','Carton_brun','papier','kraft','cartonnette','flou','mal_croppe','Carton_gris','environnement','autre_refus','plastique')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20973396 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Teint_Dans_La_Masse','metal','Carton_brun','papier','kraft','cartonnette','flou','mal_croppe','Carton_gris','environnement','autre_refus','plastique')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20973640,20973641,20973642,20973643,20973644,20973645,20973646,20973647,20973648,20973649,20973650,20973651?tags=autre_refus,Carton_brun,metal,plastique,papier,flou,cartonnette,environnement,Carton_gris,kraft,mal_croppe,Teint_Dans_La_Masse Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1340612350, 1340612307, 1340612306, 1340612251, 1340612249, 1340612173, 1340612169, 1340612077, 1340612074, 1340612030] Looping around the photos to save general results len do output : 1 /20973396. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612350', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612307', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612306', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612251', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612249', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612173', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612169', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612077', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612074', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612030', None, None, None, None, None, '2629128') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.014989614486694336 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.567234754562378 time spend to save output : 0.01525735855102539 total time spend for step 7 : 1.5824921131134033 step8:final Mon Mar 3 12:14:54 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1340612350: ('0.06606508575122308',), 1340612307: ('0.06606508575122308',), 1340612306: ('0.06606508575122308',), 1340612251: ('0.06606508575122308',), 1340612249: ('0.06606508575122308',), 1340612173: ('0.06606508575122308',), 1340612169: ('0.06606508575122308',), 1340612077: ('0.06606508575122308',), 1340612074: ('0.06606508575122308',), 1340612030: ('0.06606508575122308',)} new output for save of step final : {1340612350: ('0.06606508575122308',), 1340612307: ('0.06606508575122308',), 1340612306: ('0.06606508575122308',), 1340612251: ('0.06606508575122308',), 1340612249: ('0.06606508575122308',), 1340612173: ('0.06606508575122308',), 1340612169: ('0.06606508575122308',), 1340612077: ('0.06606508575122308',), 1340612074: ('0.06606508575122308',), 1340612030: ('0.06606508575122308',)} [1340612350, 1340612307, 1340612306, 1340612251, 1340612249, 1340612173, 1340612169, 1340612077, 1340612074, 1340612030] Looping around the photos to save general results len do output : 10 /1340612350.Didn't retrieve data . /1340612307.Didn't retrieve data . /1340612306.Didn't retrieve data . /1340612251.Didn't retrieve data . /1340612249.Didn't retrieve data . /1340612173.Didn't retrieve data . /1340612169.Didn't retrieve data . /1340612077.Didn't retrieve data . /1340612074.Didn't retrieve data . /1340612030.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612350', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612307', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612306', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612251', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612249', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612173', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612169', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612077', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612074', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612030', None, None, None, None, None, '2629128') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.012648820877075195 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.0958399772644043 time spend to save output : 0.013145208358764648 total time spend for step 8 : 0.10898518562316895 step9:velours_tree Mon Mar 3 12:14:54 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.0311279296875 time spend to save output : 3.62396240234375e-05 total time spend for step 9 : 0.031164169311523438 step10:send_mail_cod Mon Mar 3 12:14:54 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20973396_03-03-2025_12_14_54.pdf 20973640 change filename to text .change filename to text .imagette209736401741000494 20973642 imagette209736421741000495 20973643 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette209736431741000495 20973645 imagette209736451741000495 20973646 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette209736461741000495 20973649 change filename to text .change filename to text .imagette209736491741000495 20973650 imagette209736501741000495 20973641 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette209736411741000495 20973648 change filename to text .imagette209736481741000496 20973651 change filename to text .change filename to text .change filename to text .imagette209736511741000496 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20973396 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20973640,20973641,20973642,20973643,20973644,20973645,20973646,20973647,20973648,20973649,20973650,20973651?tags=autre_refus,Carton_brun,metal,plastique,papier,flou,cartonnette,environnement,Carton_gris,kraft,mal_croppe,Teint_Dans_La_Masse your option no_mail is active, we will not send the real mail to your client args[1340612350] : ('0.06606508575122308',) no score found for photo 1340612350 We are sending mail with results at report@fotonower.com args[1340612307] : ('0.06606508575122308',) no score found for photo 1340612307 We are sending mail with results at report@fotonower.com args[1340612306] : ('0.06606508575122308',) no score found for photo 1340612306 We are sending mail with results at report@fotonower.com args[1340612251] : ('0.06606508575122308',) no score found for photo 1340612251 We are sending mail with results at report@fotonower.com args[1340612249] : ('0.06606508575122308',) no score found for photo 1340612249 We are sending mail with results at report@fotonower.com args[1340612173] : ('0.06606508575122308',) no score found for photo 1340612173 We are sending mail with results at report@fotonower.com args[1340612169] : ('0.06606508575122308',) no score found for photo 1340612169 We are sending mail with results at report@fotonower.com args[1340612077] : ('0.06606508575122308',) no score found for photo 1340612077 We are sending mail with results at report@fotonower.com args[1340612074] : ('0.06606508575122308',) no score found for photo 1340612074 We are sending mail with results at report@fotonower.com args[1340612030] : ('0.06606508575122308',) no score found for photo 1340612030 We are sending mail with results at report@fotonower.com refus_total : 0.06606508575122308 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20973396 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1340611886,1340612074,1340611991,1340612027,1340612419,1340612307,1340612249,1340612350,1340612351,1340612420,1340612492,1340612501,1340612589,1340612590,1340612666,1340612667,1340612757,1340612306,1340612251,1340611669) Found this number of photos: 20 begin to download photo : 1340611886 begin to download photo : 1340612307 begin to download photo : 1340612492 begin to download photo : 1340612667 download finish for photo 1340612492 begin to download photo : 1340612501 download finish for photo 1340612307 begin to download photo : 1340612249 download finish for photo 1340611886 begin to download photo : 1340612074 download finish for photo 1340612667 begin to download photo : 1340612757 download finish for photo 1340612249 begin to download photo : 1340612350 download finish for photo 1340612501 begin to download photo : 1340612589 download finish for photo 1340612757 begin to download photo : 1340612306 download finish for photo 1340612074 begin to download photo : 1340611991 download finish for photo 1340612589 begin to download photo : 1340612590 download finish for photo 1340612306 begin to download photo : 1340612251 download finish for photo 1340611991 begin to download photo : 1340612027 download finish for photo 1340612350 begin to download photo : 1340612351 download finish for photo 1340612590 begin to download photo : 1340612666 download finish for photo 1340612251 begin to download photo : 1340611669 download finish for photo 1340612027 begin to download photo : 1340612419 download finish for photo 1340612351 begin to download photo : 1340612420 download finish for photo 1340612666 download finish for photo 1340611669 download finish for photo 1340612420 download finish for photo 1340612419 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20973396_03-03-2025_12_14_54.pdf results_Auto_P20973396_03-03-2025_12_14_54.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20973396_03-03-2025_12_14_54.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20973396','results_Auto_P20973396_03-03-2025_12_14_54.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20973396_03-03-2025_12_14_54.pdf','pdf','','0.36','0.06606508575122308') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1340612350, 1340612307, 1340612306, 1340612251, 1340612249, 1340612173, 1340612169, 1340612077, 1340612074, 1340612030] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612350', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612307', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612306', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612251', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612249', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612173', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612169', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612077', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612074', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612030', None, None, None, None, None, '2629128') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.5903940200805664 save_final save missing photos in datou_result : time spend for datou_step_exec : 4.075029373168945 time spend to save output : 0.5907049179077148 total time spend for step 10 : 4.66573429107666 step11:split_time_score Mon Mar 3 12:14:59 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('11', 30),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 03032025 20973396 Nombre de photos uploadées : 30 / 23040 (0%) 03032025 20973396 Nombre de photos taguées (types de déchets): 0 / 30 (0%) 03032025 20973396 Nombre de photos taguées (volume) : 0 / 30 (0%) elapsed_time : load_data_split_time_score 3.5762786865234375e-06 elapsed_time : order_list_meta_photo_and_scores 8.106231689453125e-06 ?????????????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.002803802490234375 elapsed_time : insert_dashboard_record_day_entry 0.023105859756469727 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.07226605924691387 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20964506_03-03-2025_08_42_01.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20964506 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20964506 AND mptpi.`type`=3726 To do Qualite : 0.07603242440997401 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20965613_03-03-2025_08_25_42.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20965613 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20965613 AND mptpi.`type`=3726 To do Qualite : 0.02575062473468879 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20965638_03-03-2025_08_15_41.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20965638 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20965638 AND mptpi.`type`=3726 To do Qualite : 0.05752190734891299 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20965639_03-03-2025_08_11_47.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20965639 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20965639 AND mptpi.`type`=3726 To do Qualite : 0.03688006233820538 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20966600_03-03-2025_09_11_26.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20966600 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20966600 AND mptpi.`type`=3726 To do Qualite : 0.02831058440182939 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20967444_03-03-2025_09_46_54.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20967444 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20967444 AND mptpi.`type`=3726 To do Qualite : 0.021500941685675855 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20967445_03-03-2025_09_40_48.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20967445 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20967445 AND mptpi.`type`=3726 To do Qualite : 0.019566236202716712 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20969599_03-03-2025_12_02_48.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20969599 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20969599 AND mptpi.`type`=3726 To do Qualite : 0.04866769047720368 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20969612_03-03-2025_10_52_27.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20969612 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20969612 AND mptpi.`type`=3726 To do Qualite : 0.027958931338686502 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20969625_03-03-2025_11_57_30.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20969625 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20969625 AND mptpi.`type`=3726 To do Qualite : 0.03740780134606926 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20969626_03-03-2025_10_43_47.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20969626 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20969626 AND mptpi.`type`=3726 To do Qualite : 0.024803821231226257 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20970847_03-03-2025_11_15_37.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20970847 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20970847 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20970848 order by id desc limit 1 Qualite : 0.045046809502109764 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20971881_03-03-2025_11_49_53.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20971881 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20971881 AND mptpi.`type`=3726 To do Qualite : 0.030539710118293865 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20973395_03-03-2025_12_14_07.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20973395 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20973395 AND mptpi.`type`=3726 To do Qualite : 0.03595774732673973 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20973396_03-03-2025_12_14_54.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20973396 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20973396 AND mptpi.`type`=3726 To do Qualite : 0.04586978746460799 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20973397_03-03-2025_12_12_35.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20973397 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20973397 AND mptpi.`type`=3726 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'03032025': {'nb_upload': 30, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1340612350, 1340612307, 1340612306, 1340612251, 1340612249, 1340612173, 1340612169, 1340612077, 1340612074, 1340612030] Looping around the photos to save general results len do output : 1 /20973396Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612350', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612307', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612306', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612251', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612249', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612173', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612169', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612077', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612074', None, None, None, None, None, '2629128') ('3459', None, None, None, None, None, None, None, '2629128') ('3459', None, '1340612030', None, None, None, None, None, '2629128') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.013335943222045898 save_final save missing photos in datou_result : time spend for datou_step_exec : 2.7858715057373047 time spend to save output : 0.013542890548706055 total time spend for step 11 : 2.7994143962860107 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 54.76user 36.80system 2:37.17elapsed 58%CPU (0avgtext+0avgdata 2735828maxresident)k 2015272inputs+45936outputs (7052major+2134813minor)pagefaults 0swaps