python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3182781 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2635391'] with mtr_portfolio_ids : ['21042321'] and first list_photo_ids : [] new path : /proc/3182781/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 2.5535013675689697 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Wed Mar 5 04:10:31 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10593 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-03-05 04:10:34.576151: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-05 04:10:34.607292: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-05 04:10:34.609018: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f760c000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-05 04:10:34.609070: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-05 04:10:34.612046: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-05 04:10:34.736559: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xc56eb10 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-05 04:10:34.736614: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-05 04:10:34.738100: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-05 04:10:34.738641: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 04:10:34.741876: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 04:10:34.745069: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-05 04:10:34.745695: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-05 04:10:34.748896: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-05 04:10:34.750005: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-05 04:10:34.754788: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-05 04:10:34.756418: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-05 04:10:34.756516: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 04:10:34.757291: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-05 04:10:34.757309: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-05 04:10:34.757319: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-05 04:10:34.758707: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9064 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-03-05 04:10:35.049954: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-05 04:10:35.050108: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 04:10:35.050164: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 04:10:35.050196: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-05 04:10:35.050226: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-05 04:10:35.050256: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-05 04:10:35.050285: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-05 04:10:35.050315: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-05 04:10:35.052458: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-05 04:10:35.053817: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-05 04:10:35.053879: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-05 04:10:35.053896: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 04:10:35.053910: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-05 04:10:35.053925: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-05 04:10:35.053939: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-05 04:10:35.053954: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-05 04:10:35.053969: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-05 04:10:35.055166: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-05 04:10:35.055206: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-05 04:10:35.055215: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-05 04:10:35.055223: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-05 04:10:35.056497: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9064 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-03-05 04:10:46.144336: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-05 04:10:46.359831: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-05 04:10:47.830911: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:47.831628: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.60G (3865470464 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:47.832309: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3478923264 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.648386: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.648494: W tensorflow/core/common_runtime/bfc_allocator.cc:311] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature. 2025-03-05 04:10:48.719640: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.719731: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 04:10:48.721110: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.721143: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 04:10:48.728459: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.728487: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.78GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 04:10:48.729213: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.729228: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.78GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 04:10:48.763405: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.763551: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 19.91MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 04:10:48.763614: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-03-05 04:10:48.764910: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.764940: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 16.00MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 04:10:48.766076: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.766111: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 16.00MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 04:10:48.775936: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.776022: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 63.85MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 04:10:48.777034: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.777074: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 63.85MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 04:10:48.778099: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.778126: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.26GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-05 04:10:48.779183: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.790054: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.790922: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.816047: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.816926: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.817750: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.818550: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.823369: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.824191: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.825003: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.825796: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.827120: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.838996: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.839797: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.851057: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.851855: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.852654: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.853443: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.854401: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-05 04:10:48.855205: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 22 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 24 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 29 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 15 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 9 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 13 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 15 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 11 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 22 Detection mask done ! Trying to reset tf kernel 3183548 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 4111 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5304 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.00637364387512207 nb_pixel_total : 152331 time to create 1 rle with new method : 0.01117849349975586 length of segment : 596 time for calcul the mask position with numpy : 0.0005671977996826172 nb_pixel_total : 27699 time to create 1 rle with old method : 0.03201007843017578 length of segment : 224 time for calcul the mask position with numpy : 0.00024509429931640625 nb_pixel_total : 11968 time to create 1 rle with old method : 0.013802289962768555 length of segment : 154 time for calcul the mask position with numpy : 0.041971683502197266 nb_pixel_total : 733538 time to create 1 rle with new method : 0.1425638198852539 length of segment : 1857 time for calcul the mask position with numpy : 0.0008180141448974609 nb_pixel_total : 35837 time to create 1 rle with old method : 0.04265403747558594 length of segment : 244 time for calcul the mask position with numpy : 0.0007174015045166016 nb_pixel_total : 16208 time to create 1 rle with old method : 0.018856048583984375 length of segment : 269 time for calcul the mask position with numpy : 0.0001468658447265625 nb_pixel_total : 1025 time to create 1 rle with old method : 0.001420736312866211 length of segment : 34 time for calcul the mask position with numpy : 0.00019288063049316406 nb_pixel_total : 3686 time to create 1 rle with old method : 0.004835605621337891 length of segment : 99 time for calcul the mask position with numpy : 0.0003046989440917969 nb_pixel_total : 11184 time to create 1 rle with old method : 0.013319730758666992 length of segment : 177 time for calcul the mask position with numpy : 8.225440979003906e-05 nb_pixel_total : 1336 time to create 1 rle with old method : 0.001811981201171875 length of segment : 47 time for calcul the mask position with numpy : 0.0001552104949951172 nb_pixel_total : 6380 time to create 1 rle with old method : 0.007918834686279297 length of segment : 90 time for calcul the mask position with numpy : 0.00029277801513671875 nb_pixel_total : 11234 time to create 1 rle with old method : 0.013271808624267578 length of segment : 153 time for calcul the mask position with numpy : 0.007208347320556641 nb_pixel_total : 369640 time to create 1 rle with new method : 0.021509885787963867 length of segment : 1207 time for calcul the mask position with numpy : 0.00066375732421875 nb_pixel_total : 36497 time to create 1 rle with old method : 0.04146170616149902 length of segment : 276 time for calcul the mask position with numpy : 0.0022203922271728516 nb_pixel_total : 135681 time to create 1 rle with old method : 0.17728614807128906 length of segment : 574 time for calcul the mask position with numpy : 0.0003509521484375 nb_pixel_total : 7755 time to create 1 rle with old method : 0.00977015495300293 length of segment : 144 time for calcul the mask position with numpy : 0.0006606578826904297 nb_pixel_total : 32621 time to create 1 rle with old method : 0.037673234939575195 length of segment : 223 time for calcul the mask position with numpy : 0.004629611968994141 nb_pixel_total : 207554 time to create 1 rle with new method : 0.01222372055053711 length of segment : 578 time for calcul the mask position with numpy : 0.0003185272216796875 nb_pixel_total : 7403 time to create 1 rle with old method : 0.009137153625488281 length of segment : 113 time for calcul the mask position with numpy : 0.0041255950927734375 nb_pixel_total : 153200 time to create 1 rle with new method : 0.019192218780517578 length of segment : 603 time for calcul the mask position with numpy : 0.023389101028442383 nb_pixel_total : 716099 time to create 1 rle with new method : 0.15604090690612793 length of segment : 1659 time for calcul the mask position with numpy : 0.0006678104400634766 nb_pixel_total : 8541 time to create 1 rle with old method : 0.01002049446105957 length of segment : 120 time for calcul the mask position with numpy : 0.00020813941955566406 nb_pixel_total : 1541 time to create 1 rle with old method : 0.0018820762634277344 length of segment : 49 time for calcul the mask position with numpy : 0.0007147789001464844 nb_pixel_total : 10526 time to create 1 rle with old method : 0.012695074081420898 length of segment : 178 time for calcul the mask position with numpy : 0.00037980079650878906 nb_pixel_total : 4667 time to create 1 rle with old method : 0.005876779556274414 length of segment : 72 time for calcul the mask position with numpy : 0.005316257476806641 nb_pixel_total : 148761 time to create 1 rle with old method : 0.1666858196258545 length of segment : 576 time for calcul the mask position with numpy : 0.002022266387939453 nb_pixel_total : 25795 time to create 1 rle with old method : 0.041104793548583984 length of segment : 345 time for calcul the mask position with numpy : 0.006145954132080078 nb_pixel_total : 132414 time to create 1 rle with old method : 0.1525423526763916 length of segment : 556 time for calcul the mask position with numpy : 0.002277851104736328 nb_pixel_total : 30028 time to create 1 rle with old method : 0.03893542289733887 length of segment : 219 time for calcul the mask position with numpy : 0.00016999244689941406 nb_pixel_total : 709 time to create 1 rle with old method : 0.0009765625 length of segment : 31 time for calcul the mask position with numpy : 0.005674123764038086 nb_pixel_total : 217768 time to create 1 rle with new method : 0.01190185546875 length of segment : 575 time for calcul the mask position with numpy : 0.0005974769592285156 nb_pixel_total : 11160 time to create 1 rle with old method : 0.013045310974121094 length of segment : 245 time for calcul the mask position with numpy : 0.0007166862487792969 nb_pixel_total : 10789 time to create 1 rle with old method : 0.013008832931518555 length of segment : 199 time for calcul the mask position with numpy : 0.0002384185791015625 nb_pixel_total : 874 time to create 1 rle with old method : 0.0011801719665527344 length of segment : 47 time for calcul the mask position with numpy : 0.0009729862213134766 nb_pixel_total : 15115 time to create 1 rle with old method : 0.01860189437866211 length of segment : 219 time for calcul the mask position with numpy : 0.004569292068481445 nb_pixel_total : 213245 time to create 1 rle with new method : 0.01149296760559082 length of segment : 573 time for calcul the mask position with numpy : 0.008573532104492188 nb_pixel_total : 245351 time to create 1 rle with new method : 0.014394760131835938 length of segment : 885 time for calcul the mask position with numpy : 0.0039386749267578125 nb_pixel_total : 86660 time to create 1 rle with old method : 0.10036778450012207 length of segment : 333 time for calcul the mask position with numpy : 0.00484466552734375 nb_pixel_total : 120407 time to create 1 rle with old method : 0.16132688522338867 length of segment : 442 time for calcul the mask position with numpy : 0.0008747577667236328 nb_pixel_total : 12579 time to create 1 rle with old method : 0.014768362045288086 length of segment : 169 time for calcul the mask position with numpy : 0.00021791458129882812 nb_pixel_total : 2926 time to create 1 rle with old method : 0.003809690475463867 length of segment : 47 time for calcul the mask position with numpy : 0.0008985996246337891 nb_pixel_total : 19876 time to create 1 rle with old method : 0.024142980575561523 length of segment : 202 time for calcul the mask position with numpy : 0.004489421844482422 nb_pixel_total : 102948 time to create 1 rle with old method : 0.11522340774536133 length of segment : 501 time for calcul the mask position with numpy : 0.0067365169525146484 nb_pixel_total : 137061 time to create 1 rle with old method : 0.15282917022705078 length of segment : 660 time for calcul the mask position with numpy : 0.0010571479797363281 nb_pixel_total : 14942 time to create 1 rle with old method : 0.017618656158447266 length of segment : 273 time for calcul the mask position with numpy : 0.004882097244262695 nb_pixel_total : 162887 time to create 1 rle with new method : 0.008983135223388672 length of segment : 391 time for calcul the mask position with numpy : 0.00023865699768066406 nb_pixel_total : 2756 time to create 1 rle with old method : 0.003309965133666992 length of segment : 74 time for calcul the mask position with numpy : 0.007922649383544922 nb_pixel_total : 201551 time to create 1 rle with new method : 0.013865947723388672 length of segment : 655 time for calcul the mask position with numpy : 0.0024597644805908203 nb_pixel_total : 75868 time to create 1 rle with old method : 0.08696579933166504 length of segment : 306 time for calcul the mask position with numpy : 0.002390623092651367 nb_pixel_total : 47788 time to create 1 rle with old method : 0.053635597229003906 length of segment : 236 time for calcul the mask position with numpy : 8.368492126464844e-05 nb_pixel_total : 1681 time to create 1 rle with old method : 0.0020813941955566406 length of segment : 37 time for calcul the mask position with numpy : 0.002994537353515625 nb_pixel_total : 86161 time to create 1 rle with old method : 0.09782791137695312 length of segment : 604 time for calcul the mask position with numpy : 0.0002551078796386719 nb_pixel_total : 2039 time to create 1 rle with old method : 0.002529621124267578 length of segment : 72 time for calcul the mask position with numpy : 0.004454851150512695 nb_pixel_total : 112386 time to create 1 rle with old method : 0.12529373168945312 length of segment : 404 time for calcul the mask position with numpy : 0.0003352165222167969 nb_pixel_total : 2178 time to create 1 rle with old method : 0.0027921199798583984 length of segment : 85 time for calcul the mask position with numpy : 0.0005283355712890625 nb_pixel_total : 7398 time to create 1 rle with old method : 0.008778095245361328 length of segment : 151 time for calcul the mask position with numpy : 0.0022382736206054688 nb_pixel_total : 50500 time to create 1 rle with old method : 0.05933856964111328 length of segment : 329 time for calcul the mask position with numpy : 0.008435964584350586 nb_pixel_total : 241015 time to create 1 rle with new method : 0.036583662033081055 length of segment : 678 time for calcul the mask position with numpy : 0.004060029983520508 nb_pixel_total : 120430 time to create 1 rle with old method : 0.13579988479614258 length of segment : 387 time for calcul the mask position with numpy : 0.0012562274932861328 nb_pixel_total : 18387 time to create 1 rle with old method : 0.021709680557250977 length of segment : 156 time for calcul the mask position with numpy : 0.004436969757080078 nb_pixel_total : 164430 time to create 1 rle with new method : 0.011585235595703125 length of segment : 416 time for calcul the mask position with numpy : 0.0016131401062011719 nb_pixel_total : 4706 time to create 1 rle with old method : 0.0060884952545166016 length of segment : 120 time for calcul the mask position with numpy : 0.0011467933654785156 nb_pixel_total : 2563 time to create 1 rle with old method : 0.003415822982788086 length of segment : 288 time for calcul the mask position with numpy : 0.005386829376220703 nb_pixel_total : 115585 time to create 1 rle with old method : 0.1302933692932129 length of segment : 428 time for calcul the mask position with numpy : 0.0023512840270996094 nb_pixel_total : 104537 time to create 1 rle with old method : 0.1201784610748291 length of segment : 362 time for calcul the mask position with numpy : 0.00028777122497558594 nb_pixel_total : 6444 time to create 1 rle with old method : 0.007886409759521484 length of segment : 94 time for calcul the mask position with numpy : 0.0024671554565429688 nb_pixel_total : 83892 time to create 1 rle with old method : 0.09637784957885742 length of segment : 348 time for calcul the mask position with numpy : 0.003047466278076172 nb_pixel_total : 87369 time to create 1 rle with old method : 0.09716129302978516 length of segment : 458 time for calcul the mask position with numpy : 0.0010976791381835938 nb_pixel_total : 33028 time to create 1 rle with old method : 0.03825497627258301 length of segment : 206 time for calcul the mask position with numpy : 0.00455021858215332 nb_pixel_total : 165137 time to create 1 rle with new method : 0.008436918258666992 length of segment : 421 time for calcul the mask position with numpy : 0.00022554397583007812 nb_pixel_total : 3746 time to create 1 rle with old method : 0.00456547737121582 length of segment : 131 time for calcul the mask position with numpy : 0.003964424133300781 nb_pixel_total : 104064 time to create 1 rle with old method : 0.13623523712158203 length of segment : 532 time for calcul the mask position with numpy : 0.0006728172302246094 nb_pixel_total : 19551 time to create 1 rle with old method : 0.0323023796081543 length of segment : 149 time for calcul the mask position with numpy : 0.005557060241699219 nb_pixel_total : 267677 time to create 1 rle with new method : 0.01191401481628418 length of segment : 775 time for calcul the mask position with numpy : 0.0005033016204833984 nb_pixel_total : 7087 time to create 1 rle with old method : 0.008461236953735352 length of segment : 114 time for calcul the mask position with numpy : 0.0005481243133544922 nb_pixel_total : 10152 time to create 1 rle with old method : 0.014905929565429688 length of segment : 113 time for calcul the mask position with numpy : 0.001964569091796875 nb_pixel_total : 21730 time to create 1 rle with old method : 0.02860736846923828 length of segment : 307 time for calcul the mask position with numpy : 0.0010404586791992188 nb_pixel_total : 20372 time to create 1 rle with old method : 0.023535728454589844 length of segment : 164 time for calcul the mask position with numpy : 0.0012476444244384766 nb_pixel_total : 54795 time to create 1 rle with old method : 0.06300473213195801 length of segment : 209 time for calcul the mask position with numpy : 0.0020999908447265625 nb_pixel_total : 85474 time to create 1 rle with old method : 0.09918904304504395 length of segment : 465 time for calcul the mask position with numpy : 0.0011713504791259766 nb_pixel_total : 33153 time to create 1 rle with old method : 0.05409502983093262 length of segment : 286 time for calcul the mask position with numpy : 0.002958059310913086 nb_pixel_total : 101551 time to create 1 rle with old method : 0.13886404037475586 length of segment : 705 time for calcul the mask position with numpy : 0.0013320446014404297 nb_pixel_total : 43853 time to create 1 rle with old method : 0.04995226860046387 length of segment : 237 time for calcul the mask position with numpy : 0.0015838146209716797 nb_pixel_total : 39777 time to create 1 rle with old method : 0.04699850082397461 length of segment : 330 time for calcul the mask position with numpy : 0.0009751319885253906 nb_pixel_total : 31346 time to create 1 rle with old method : 0.03901243209838867 length of segment : 311 time for calcul the mask position with numpy : 0.0007181167602539062 nb_pixel_total : 10518 time to create 1 rle with old method : 0.011879920959472656 length of segment : 263 time for calcul the mask position with numpy : 0.0019690990447998047 nb_pixel_total : 61636 time to create 1 rle with old method : 0.08738207817077637 length of segment : 288 time for calcul the mask position with numpy : 0.0017547607421875 nb_pixel_total : 82376 time to create 1 rle with old method : 0.09336328506469727 length of segment : 758 time for calcul the mask position with numpy : 0.0008838176727294922 nb_pixel_total : 44406 time to create 1 rle with old method : 0.05258464813232422 length of segment : 158 time for calcul the mask position with numpy : 0.0009908676147460938 nb_pixel_total : 27980 time to create 1 rle with old method : 0.03254866600036621 length of segment : 177 time for calcul the mask position with numpy : 0.0002987384796142578 nb_pixel_total : 5078 time to create 1 rle with old method : 0.006337404251098633 length of segment : 65 time for calcul the mask position with numpy : 0.000993967056274414 nb_pixel_total : 54448 time to create 1 rle with old method : 0.06677961349487305 length of segment : 312 time for calcul the mask position with numpy : 0.0007190704345703125 nb_pixel_total : 16780 time to create 1 rle with old method : 0.01944255828857422 length of segment : 234 time for calcul the mask position with numpy : 0.0005292892456054688 nb_pixel_total : 40468 time to create 1 rle with old method : 0.047850608825683594 length of segment : 146 time for calcul the mask position with numpy : 0.003860950469970703 nb_pixel_total : 158601 time to create 1 rle with new method : 0.007055759429931641 length of segment : 724 time for calcul the mask position with numpy : 0.0005242824554443359 nb_pixel_total : 15247 time to create 1 rle with old method : 0.0182192325592041 length of segment : 138 time for calcul the mask position with numpy : 0.00027942657470703125 nb_pixel_total : 9130 time to create 1 rle with old method : 0.010891437530517578 length of segment : 91 time for calcul the mask position with numpy : 0.27547216415405273 nb_pixel_total : 2828157 time to create 1 rle with new method : 0.39208173751831055 length of segment : 3762 time for calcul the mask position with numpy : 0.00011587142944335938 nb_pixel_total : 2010 time to create 1 rle with old method : 0.002508401870727539 length of segment : 57 time for calcul the mask position with numpy : 0.12282896041870117 nb_pixel_total : 1516617 time to create 1 rle with new method : 0.11671113967895508 length of segment : 1724 time for calcul the mask position with numpy : 0.01107931137084961 nb_pixel_total : 473930 time to create 1 rle with new method : 0.09659957885742188 length of segment : 1294 time for calcul the mask position with numpy : 0.001279592514038086 nb_pixel_total : 88386 time to create 1 rle with old method : 0.09803318977355957 length of segment : 364 time for calcul the mask position with numpy : 0.002666950225830078 nb_pixel_total : 83713 time to create 1 rle with old method : 0.11564064025878906 length of segment : 505 time for calcul the mask position with numpy : 0.0015017986297607422 nb_pixel_total : 54333 time to create 1 rle with old method : 0.061292409896850586 length of segment : 334 time for calcul the mask position with numpy : 0.2719278335571289 nb_pixel_total : 1209127 time to create 1 rle with new method : 0.13797664642333984 length of segment : 2171 time for calcul the mask position with numpy : 0.00304412841796875 nb_pixel_total : 201528 time to create 1 rle with new method : 0.006180286407470703 length of segment : 510 time for calcul the mask position with numpy : 0.0009849071502685547 nb_pixel_total : 36886 time to create 1 rle with old method : 0.04200172424316406 length of segment : 234 time for calcul the mask position with numpy : 0.0004837512969970703 nb_pixel_total : 14687 time to create 1 rle with old method : 0.01775193214416504 length of segment : 188 time for calcul the mask position with numpy : 0.00014138221740722656 nb_pixel_total : 2572 time to create 1 rle with old method : 0.003137350082397461 length of segment : 73 time for calcul the mask position with numpy : 0.0013916492462158203 nb_pixel_total : 19770 time to create 1 rle with old method : 0.023992538452148438 length of segment : 698 time for calcul the mask position with numpy : 0.02746748924255371 nb_pixel_total : 885547 time to create 1 rle with new method : 0.23926854133605957 length of segment : 2970 time for calcul the mask position with numpy : 0.0011646747589111328 nb_pixel_total : 52966 time to create 1 rle with old method : 0.06132364273071289 length of segment : 442 time for calcul the mask position with numpy : 0.00016927719116210938 nb_pixel_total : 3275 time to create 1 rle with old method : 0.003984212875366211 length of segment : 80 time for calcul the mask position with numpy : 0.0035622119903564453 nb_pixel_total : 138782 time to create 1 rle with old method : 0.17634892463684082 length of segment : 479 time for calcul the mask position with numpy : 0.002413511276245117 nb_pixel_total : 91797 time to create 1 rle with old method : 0.11494302749633789 length of segment : 379 time for calcul the mask position with numpy : 0.0015981197357177734 nb_pixel_total : 68223 time to create 1 rle with old method : 0.0783536434173584 length of segment : 501 time for calcul the mask position with numpy : 0.00045561790466308594 nb_pixel_total : 12899 time to create 1 rle with old method : 0.015361309051513672 length of segment : 94 time for calcul the mask position with numpy : 0.01138758659362793 nb_pixel_total : 223216 time to create 1 rle with new method : 0.015221118927001953 length of segment : 1776 time for calcul the mask position with numpy : 0.00035381317138671875 nb_pixel_total : 5635 time to create 1 rle with old method : 0.006761789321899414 length of segment : 105 time for calcul the mask position with numpy : 0.0008144378662109375 nb_pixel_total : 18739 time to create 1 rle with old method : 0.022774219512939453 length of segment : 173 time for calcul the mask position with numpy : 0.0022950172424316406 nb_pixel_total : 95475 time to create 1 rle with old method : 0.1089479923248291 length of segment : 298 time for calcul the mask position with numpy : 0.002196073532104492 nb_pixel_total : 79806 time to create 1 rle with old method : 0.08936405181884766 length of segment : 312 time for calcul the mask position with numpy : 0.00054168701171875 nb_pixel_total : 14116 time to create 1 rle with old method : 0.019207000732421875 length of segment : 161 time for calcul the mask position with numpy : 0.003125905990600586 nb_pixel_total : 117995 time to create 1 rle with old method : 0.1351480484008789 length of segment : 445 time for calcul the mask position with numpy : 0.0005376338958740234 nb_pixel_total : 13469 time to create 1 rle with old method : 0.020279407501220703 length of segment : 154 time for calcul the mask position with numpy : 0.06368541717529297 nb_pixel_total : 1133140 time to create 1 rle with new method : 0.23218107223510742 length of segment : 2484 time for calcul the mask position with numpy : 0.011776447296142578 nb_pixel_total : 208040 time to create 1 rle with new method : 0.01357889175415039 length of segment : 682 time for calcul the mask position with numpy : 0.0004138946533203125 nb_pixel_total : 16243 time to create 1 rle with old method : 0.01800680160522461 length of segment : 152 time for calcul the mask position with numpy : 0.00040793418884277344 nb_pixel_total : 10957 time to create 1 rle with old method : 0.015174627304077148 length of segment : 127 time for calcul the mask position with numpy : 0.0001666545867919922 nb_pixel_total : 5845 time to create 1 rle with old method : 0.0067255496978759766 length of segment : 74 time for calcul the mask position with numpy : 0.008974313735961914 nb_pixel_total : 330645 time to create 1 rle with new method : 0.02729344367980957 length of segment : 1027 time for calcul the mask position with numpy : 0.0022945404052734375 nb_pixel_total : 102482 time to create 1 rle with old method : 0.11425900459289551 length of segment : 437 time for calcul the mask position with numpy : 0.0007510185241699219 nb_pixel_total : 30530 time to create 1 rle with old method : 0.044095754623413086 length of segment : 149 time for calcul the mask position with numpy : 0.002115488052368164 nb_pixel_total : 38193 time to create 1 rle with old method : 0.04638099670410156 length of segment : 241 time for calcul the mask position with numpy : 0.0004184246063232422 nb_pixel_total : 10735 time to create 1 rle with old method : 0.012683391571044922 length of segment : 130 time for calcul the mask position with numpy : 0.0316615104675293 nb_pixel_total : 732928 time to create 1 rle with new method : 0.08644461631774902 length of segment : 1517 time for calcul the mask position with numpy : 0.02535223960876465 nb_pixel_total : 690744 time to create 1 rle with new method : 0.10560202598571777 length of segment : 1066 time for calcul the mask position with numpy : 0.0022573471069335938 nb_pixel_total : 20604 time to create 1 rle with old method : 0.024579763412475586 length of segment : 453 time for calcul the mask position with numpy : 0.002087831497192383 nb_pixel_total : 76044 time to create 1 rle with old method : 0.08640623092651367 length of segment : 649 time for calcul the mask position with numpy : 0.007338285446166992 nb_pixel_total : 105470 time to create 1 rle with old method : 0.12106037139892578 length of segment : 963 time for calcul the mask position with numpy : 0.003253459930419922 nb_pixel_total : 95601 time to create 1 rle with old method : 0.11204171180725098 length of segment : 413 time for calcul the mask position with numpy : 0.00023102760314941406 nb_pixel_total : 1227 time to create 1 rle with old method : 0.0015192031860351562 length of segment : 56 time for calcul the mask position with numpy : 0.014099836349487305 nb_pixel_total : 419080 time to create 1 rle with new method : 0.7180140018463135 length of segment : 1074 time spent for convertir_results : 21.767844676971436 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 143 chid ids of type : 3663 Number RLEs to save : 64702 save missing photos in datou_result : time spend for datou_step_exec : 75.16749119758606 time spend to save output : 16.024236917495728 total time spend for step 1 : 91.19172811508179 step2:crop_condition Wed Mar 5 04:12:03 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 143 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 15 About to insert : list_path_to_insert length 15 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 11 About to insert : list_path_to_insert length 11 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 10 About to insert : list_path_to_insert length 10 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 37 About to insert : list_path_to_insert length 37 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1340898539, 1340898531, 1340898528, 1340898525, 1340898409, 1340898402, 1340898397, 1340898304, 1340898086, 1340898002] Looping around the photos to save general results len do output : 86 /-3693976083Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976102Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976107Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976104Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976120Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976121Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976136Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976151Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976149Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976150Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976153Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976147Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976174Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976200Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976203Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976085Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976099Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976115Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976119Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976110Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976134Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976156Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976165Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976197Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976198Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976186Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976100Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976096Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976113Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976128Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976140Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976160Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976172Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976169Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976184Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976223Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976098Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976108Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976118Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976114Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976143Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976145Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976125Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976141Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976131Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976126Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976146Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976132Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976139Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976152Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976154Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976155Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976166Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976167Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976164Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976162Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976170Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976177Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976178Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976188Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976192Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976194Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976206Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976205Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976204Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976209Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976211Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976222Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976221Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976214Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976220Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976210Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976216Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976091Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976084Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976106Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976130Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976163Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976171Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976189Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976215Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976212Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976095Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976097Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976196Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3693976176Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898539', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898531', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898528', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898525', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898409', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898402', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898397', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898304', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898086', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898002', None, None, None, None, None, '2635391') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 268 time used for this insertion : 2.378880500793457 save_final save missing photos in datou_result : time spend for datou_step_exec : 46.59722137451172 time spend to save output : 2.4292640686035156 total time spend for step 2 : 49.026485443115234 step3:thcl Wed Mar 5 04:12:52 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.018881797790527344 time to convert the images to numpy array : 0.46421003341674805 time to import caffe and check if the image exist : 0.011070966720581055 time to convert the images to numpy array : 0.5007138252258301 time to import caffe and check if the image exist : 0.014686346054077148 time to convert the images to numpy array : 0.5067422389984131 time to import caffe and check if the image exist : 0.027240514755249023 time to convert the images to numpy array : 0.6392102241516113 time to import caffe and check if the image exist : 0.022414207458496094 time to convert the images to numpy array : 0.6442584991455078 time to import caffe and check if the image exist : 0.014059782028198242 time to convert the images to numpy array : 0.6608672142028809 time to import caffe and check if the image exist : 0.02018570899963379 time to convert the images to numpy array : 0.6567401885986328 time to import caffe and check if the image exist : 0.017165184020996094 time to convert the images to numpy array : 0.6622560024261475 time to import caffe and check if the image exist : 0.017400503158569336 time to convert the images to numpy array : 0.6653814315795898 time to import caffe and check if the image exist : 0.024581432342529297 time to convert the images to numpy array : 0.6824235916137695 total time to convert the images to numpy array : 1.118049144744873 list photo_ids error: [] list photo_ids correct : [-3693976216, -3693976091, -3693976084, -3693976106, -3693976130, -3693976163, -3693976171, -3693976189, -3693976215, -3693976212, -3693976095, -3693976097, -3693976196, -3693976176, -3693976205, -3693976204, -3693976209, -3693976211, -3693976222, -3693976221, -3693976214, -3693976220, -3693976210, -3693976083, -3693976102, -3693976107, -3693976104, -3693976120, -3693976121, -3693976136, -3693976151, -3693976149, -3693976150, -3693976153, -3693976147, -3693976174, -3693976200, -3693976203, -3693976085, -3693976099, -3693976115, -3693976098, -3693976108, -3693976118, -3693976114, -3693976143, -3693976145, -3693976125, -3693976141, -3693976131, -3693976126, -3693976146, -3693976132, -3693976139, -3693976152, -3693976154, -3693976155, -3693976166, -3693976167, -3693976164, -3693976162, -3693976170, -3693976177, -3693976178, -3693976188, -3693976192, -3693976194, -3693976206, -3693976096, -3693976113, -3693976128, -3693976140, -3693976160, -3693976172, -3693976169, -3693976184, -3693976223, -3693976119, -3693976110, -3693976134, -3693976156, -3693976165, -3693976197, -3693976198, -3693976186, -3693976100] number of photos to traite : 86 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5304 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5085 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.2019319534301758 time used to do the prediction : 0.2521491050720215 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5083 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5083 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.271174669265747 time used to do the prediction : 0.17650175094604492 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 2.3365020751953125e-05 save missing photos in datou_result : time spend for datou_step_exec : 12.132307529449463 time spend to save output : 0.0006930828094482422 total time spend for step 3 : 12.133000612258911 step4:merge_mask_thcl_custom Wed Mar 5 04:13:04 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 143 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1340898539, 1340898531, 1340898528, 1340898525, 1340898409, 1340898402, 1340898397, 1340898304, 1340898086, 1340898002] Looping around the photos to save general results len do output : 10 /1340898539Didn't retrieve data .Didn't retrieve data . /1340898531Didn't retrieve data .Didn't retrieve data . /1340898528Didn't retrieve data .Didn't retrieve data . /1340898525Didn't retrieve data .Didn't retrieve data . /1340898409Didn't retrieve data .Didn't retrieve data . /1340898402Didn't retrieve data .Didn't retrieve data . /1340898397Didn't retrieve data .Didn't retrieve data . /1340898304Didn't retrieve data .Didn't retrieve data . /1340898086Didn't retrieve data .Didn't retrieve data . /1340898002Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898539', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898531', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898528', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898525', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898409', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898402', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898397', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898304', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898086', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898002', None, None, None, None, None, '2635391') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.014030218124389648 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.059212684631347656 time spend to save output : 0.014608144760131836 total time spend for step 4 : 0.07382082939147949 step5:rle_unique_nms_with_priority Wed Mar 5 04:13:04 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 16 nb_hashtags : 6 time to prepare the origin masks : 11.011404514312744 time for calcul the mask position with numpy : 2.004321575164795 nb_pixel_total : 5295012 time to create 1 rle with new method : 1.008420705795288 time for calcul the mask position with numpy : 0.04527115821838379 nb_pixel_total : 358752 time to create 1 rle with new method : 2.169275999069214 time for calcul the mask position with numpy : 0.036887168884277344 nb_pixel_total : 900 time to create 1 rle with old method : 0.0010826587677001953 time for calcul the mask position with numpy : 0.03561806678771973 nb_pixel_total : 32544 time to create 1 rle with old method : 0.037899017333984375 time for calcul the mask position with numpy : 0.030816316604614258 nb_pixel_total : 7727 time to create 1 rle with old method : 0.008699178695678711 time for calcul the mask position with numpy : 0.032918691635131836 nb_pixel_total : 36418 time to create 1 rle with old method : 0.041008710861206055 time for calcul the mask position with numpy : 0.033266305923461914 nb_pixel_total : 148384 time to create 1 rle with old method : 0.15958738327026367 time for calcul the mask position with numpy : 0.03831291198730469 nb_pixel_total : 1325 time to create 1 rle with old method : 0.002439260482788086 time for calcul the mask position with numpy : 0.03871750831604004 nb_pixel_total : 206907 time to create 1 rle with new method : 0.7488770484924316 time for calcul the mask position with numpy : 0.02956533432006836 nb_pixel_total : 3646 time to create 1 rle with old method : 0.004126071929931641 time for calcul the mask position with numpy : 0.02947402000427246 nb_pixel_total : 1020 time to create 1 rle with old method : 0.0012743473052978516 time for calcul the mask position with numpy : 0.028712987899780273 nb_pixel_total : 16200 time to create 1 rle with old method : 0.01816415786743164 time for calcul the mask position with numpy : 0.038149118423461914 nb_pixel_total : 27646 time to create 1 rle with old method : 0.030765295028686523 time for calcul the mask position with numpy : 0.039679765701293945 nb_pixel_total : 35539 time to create 1 rle with old method : 0.03992199897766113 time for calcul the mask position with numpy : 0.043829917907714844 nb_pixel_total : 731583 time to create 1 rle with new method : 0.649212121963501 time for calcul the mask position with numpy : 0.03426051139831543 nb_pixel_total : 11158 time to create 1 rle with old method : 0.013020515441894531 time for calcul the mask position with numpy : 0.037264108657836914 nb_pixel_total : 135479 time to create 1 rle with old method : 0.15598273277282715 create new chi : 7.780634164810181 time to delete rle : 0.015058040618896484 batch 1 Loaded 17 chid ids of type : 3726 Number RLEs to save : 15041 TO DO : save crop sub photo not yet done ! save time : 0.9700822830200195 nb_obj : 18 nb_hashtags : 6 time to prepare the origin masks : 13.50843095779419 time for calcul the mask position with numpy : 0.5390503406524658 nb_pixel_total : 5491709 time to create 1 rle with new method : 1.068997859954834 time for calcul the mask position with numpy : 0.045798301696777344 nb_pixel_total : 1467 time to create 1 rle with old method : 0.002997875213623047 time for calcul the mask position with numpy : 0.04702329635620117 nb_pixel_total : 15060 time to create 1 rle with old method : 0.017740249633789062 time for calcul the mask position with numpy : 0.04010629653930664 nb_pixel_total : 7298 time to create 1 rle with old method : 0.008303642272949219 time for calcul the mask position with numpy : 0.04463815689086914 nb_pixel_total : 870 time to create 1 rle with old method : 0.0018939971923828125 time for calcul the mask position with numpy : 0.04022645950317383 nb_pixel_total : 694 time to create 1 rle with old method : 0.0008466243743896484 time for calcul the mask position with numpy : 0.03885674476623535 nb_pixel_total : 10761 time to create 1 rle with old method : 0.011946678161621094 time for calcul the mask position with numpy : 0.040902137756347656 nb_pixel_total : 217374 time to create 1 rle with new method : 1.0289087295532227 time for calcul the mask position with numpy : 0.04340076446533203 nb_pixel_total : 4622 time to create 1 rle with old method : 0.005370140075683594 time for calcul the mask position with numpy : 0.02538776397705078 nb_pixel_total : 34018 time to create 1 rle with old method : 0.03786420822143555 time for calcul the mask position with numpy : 0.02828669548034668 nb_pixel_total : 147775 time to create 1 rle with old method : 0.16517210006713867 time for calcul the mask position with numpy : 0.026885509490966797 nb_pixel_total : 8478 time to create 1 rle with old method : 0.010030508041381836 time for calcul the mask position with numpy : 0.027906179428100586 nb_pixel_total : 132050 time to create 1 rle with old method : 0.14537334442138672 time for calcul the mask position with numpy : 0.024236440658569336 nb_pixel_total : 1529 time to create 1 rle with old method : 0.001741170883178711 time for calcul the mask position with numpy : 0.03819847106933594 nb_pixel_total : 714901 time to create 1 rle with new method : 0.9786868095397949 time for calcul the mask position with numpy : 0.045024871826171875 nb_pixel_total : 459 time to create 1 rle with old method : 0.0010962486267089844 time for calcul the mask position with numpy : 0.0420382022857666 nb_pixel_total : 10728 time to create 1 rle with old method : 0.011932373046875 time for calcul the mask position with numpy : 0.039438486099243164 nb_pixel_total : 7374 time to create 1 rle with old method : 0.009111404418945312 time for calcul the mask position with numpy : 0.04231595993041992 nb_pixel_total : 243073 time to create 1 rle with new method : 0.7824394702911377 create new chi : 5.639892578125 time to delete rle : 0.001621246337890625 batch 1 Loaded 19 chid ids of type : 3726 Number RLEs to save : 14080 TO DO : save crop sub photo not yet done ! save time : 3.6632115840911865 nb_obj : 25 nb_hashtags : 9 time to prepare the origin masks : 18.065221071243286 time for calcul the mask position with numpy : 0.42761826515197754 nb_pixel_total : 5733321 time to create 1 rle with new method : 0.4902513027191162 time for calcul the mask position with numpy : 0.03484368324279785 nb_pixel_total : 88730 time to create 1 rle with old method : 0.09896445274353027 time for calcul the mask position with numpy : 0.03493523597717285 nb_pixel_total : 50957 time to create 1 rle with old method : 0.05707263946533203 time for calcul the mask position with numpy : 0.035448551177978516 nb_pixel_total : 136837 time to create 1 rle with old method : 0.16280651092529297 time for calcul the mask position with numpy : 0.03643679618835449 nb_pixel_total : 1183 time to create 1 rle with old method : 0.00189208984375 time for calcul the mask position with numpy : 0.034221649169921875 nb_pixel_total : 4641 time to create 1 rle with old method : 0.005584716796875 time for calcul the mask position with numpy : 0.03461432456970215 nb_pixel_total : 5744 time to create 1 rle with old method : 0.08126688003540039 time for calcul the mask position with numpy : 0.036827802658081055 nb_pixel_total : 115421 time to create 1 rle with old method : 0.14632153511047363 time for calcul the mask position with numpy : 0.039366722106933594 nb_pixel_total : 101643 time to create 1 rle with old method : 0.11861705780029297 time for calcul the mask position with numpy : 0.03391623497009277 nb_pixel_total : 2168 time to create 1 rle with old method : 0.002602100372314453 time for calcul the mask position with numpy : 0.034921884536743164 nb_pixel_total : 162433 time to create 1 rle with new method : 0.9694666862487793 time for calcul the mask position with numpy : 0.03641486167907715 nb_pixel_total : 2025 time to create 1 rle with old method : 0.002854585647583008 time for calcul the mask position with numpy : 0.03441190719604492 nb_pixel_total : 1665 time to create 1 rle with old method : 0.0019099712371826172 time for calcul the mask position with numpy : 0.03366351127624512 nb_pixel_total : 75541 time to create 1 rle with old method : 0.08827733993530273 time for calcul the mask position with numpy : 0.03653454780578613 nb_pixel_total : 86329 time to create 1 rle with old method : 0.09525871276855469 time for calcul the mask position with numpy : 0.03284764289855957 nb_pixel_total : 50511 time to create 1 rle with old method : 0.0560150146484375 time for calcul the mask position with numpy : 0.030833005905151367 nb_pixel_total : 47569 time to create 1 rle with old method : 0.052092552185058594 time for calcul the mask position with numpy : 0.03515934944152832 nb_pixel_total : 201278 time to create 1 rle with new method : 0.5759563446044922 time for calcul the mask position with numpy : 0.033043622970581055 nb_pixel_total : 2749 time to create 1 rle with old method : 0.0030066967010498047 time for calcul the mask position with numpy : 0.03664255142211914 nb_pixel_total : 14873 time to create 1 rle with old method : 0.01609969139099121 time for calcul the mask position with numpy : 0.03307819366455078 nb_pixel_total : 19864 time to create 1 rle with old method : 0.02111983299255371 time for calcul the mask position with numpy : 0.03307938575744629 nb_pixel_total : 2910 time to create 1 rle with old method : 0.0032396316528320312 time for calcul the mask position with numpy : 0.03320670127868652 nb_pixel_total : 6764 time to create 1 rle with old method : 0.00802755355834961 time for calcul the mask position with numpy : 0.03015923500061035 nb_pixel_total : 12524 time to create 1 rle with old method : 0.013991594314575195 time for calcul the mask position with numpy : 0.030090808868408203 nb_pixel_total : 120029 time to create 1 rle with old method : 0.13279175758361816 time for calcul the mask position with numpy : 0.028948068618774414 nb_pixel_total : 2531 time to create 1 rle with old method : 0.0029747486114501953 create new chi : 4.573798418045044 time to delete rle : 0.0015931129455566406 batch 1 Loaded 26 chid ids of type : 3726 Number RLEs to save : 16586 TO DO : save crop sub photo not yet done ! save time : 5.17082405090332 nb_obj : 13 nb_hashtags : 5 time to prepare the origin masks : 12.28829836845398 time for calcul the mask position with numpy : 0.6079897880554199 nb_pixel_total : 6248645 time to create 1 rle with new method : 0.9197378158569336 time for calcul the mask position with numpy : 0.03841352462768555 nb_pixel_total : 932 time to create 1 rle with old method : 0.0018923282623291016 time for calcul the mask position with numpy : 0.02537393569946289 nb_pixel_total : 164780 time to create 1 rle with new method : 0.6218018531799316 time for calcul the mask position with numpy : 0.029339313507080078 nb_pixel_total : 17738 time to create 1 rle with old method : 0.020792722702026367 time for calcul the mask position with numpy : 0.036185503005981445 nb_pixel_total : 19423 time to create 1 rle with old method : 0.023415803909301758 time for calcul the mask position with numpy : 0.035269737243652344 nb_pixel_total : 1798 time to create 1 rle with old method : 0.0022928714752197266 time for calcul the mask position with numpy : 0.03551459312438965 nb_pixel_total : 10075 time to create 1 rle with old method : 0.01150655746459961 time for calcul the mask position with numpy : 0.03477835655212402 nb_pixel_total : 83564 time to create 1 rle with old method : 0.09784364700317383 time for calcul the mask position with numpy : 0.036038875579833984 nb_pixel_total : 268698 time to create 1 rle with new method : 0.6178426742553711 time for calcul the mask position with numpy : 0.03541445732116699 nb_pixel_total : 104327 time to create 1 rle with old method : 0.12279415130615234 time for calcul the mask position with numpy : 0.030863046646118164 nb_pixel_total : 32987 time to create 1 rle with old method : 0.03801846504211426 time for calcul the mask position with numpy : 0.03340554237365723 nb_pixel_total : 87128 time to create 1 rle with old method : 0.09475183486938477 time for calcul the mask position with numpy : 0.030570507049560547 nb_pixel_total : 6430 time to create 1 rle with old method : 0.006970643997192383 time for calcul the mask position with numpy : 0.0314936637878418 nb_pixel_total : 3715 time to create 1 rle with old method : 0.004155397415161133 create new chi : 3.719468116760254 time to delete rle : 0.001367807388305664 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 9227 TO DO : save crop sub photo not yet done ! save time : 1.4086060523986816 nb_obj : 7 nb_hashtags : 6 time to prepare the origin masks : 6.6588850021362305 time for calcul the mask position with numpy : 0.4906318187713623 nb_pixel_total : 6672439 time to create 1 rle with new method : 0.6626818180084229 time for calcul the mask position with numpy : 0.02087712287902832 nb_pixel_total : 31865 time to create 1 rle with old method : 0.03467845916748047 time for calcul the mask position with numpy : 0.020762205123901367 nb_pixel_total : 77140 time to create 1 rle with old method : 0.09568405151367188 time for calcul the mask position with numpy : 0.02338266372680664 nb_pixel_total : 43687 time to create 1 rle with old method : 0.05053305625915527 time for calcul the mask position with numpy : 0.022622108459472656 nb_pixel_total : 29475 time to create 1 rle with old method : 0.033782958984375 time for calcul the mask position with numpy : 0.02102828025817871 nb_pixel_total : 101342 time to create 1 rle with old method : 0.10730314254760742 time for calcul the mask position with numpy : 0.02126002311706543 nb_pixel_total : 39655 time to create 1 rle with old method : 0.04580879211425781 time for calcul the mask position with numpy : 0.02365875244140625 nb_pixel_total : 54637 time to create 1 rle with old method : 0.07879400253295898 create new chi : 1.792154312133789 time to delete rle : 0.0007083415985107422 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 6763 TO DO : save crop sub photo not yet done ! save time : 1.6784143447875977 nb_obj : 11 nb_hashtags : 5 time to prepare the origin masks : 7.373862981796265 time for calcul the mask position with numpy : 0.48893165588378906 nb_pixel_total : 6630778 time to create 1 rle with new method : 0.5788524150848389 time for calcul the mask position with numpy : 0.020390987396240234 nb_pixel_total : 15137 time to create 1 rle with old method : 0.016230344772338867 time for calcul the mask position with numpy : 0.03483152389526367 nb_pixel_total : 157346 time to create 1 rle with new method : 0.4766418933868408 time for calcul the mask position with numpy : 0.023932456970214844 nb_pixel_total : 27856 time to create 1 rle with old method : 0.0305483341217041 time for calcul the mask position with numpy : 0.021741867065429688 nb_pixel_total : 100 time to create 1 rle with old method : 0.0004127025604248047 time for calcul the mask position with numpy : 0.024980545043945312 nb_pixel_total : 61416 time to create 1 rle with old method : 0.06984519958496094 time for calcul the mask position with numpy : 0.020755290985107422 nb_pixel_total : 16663 time to create 1 rle with old method : 0.018082618713378906 time for calcul the mask position with numpy : 0.034342050552368164 nb_pixel_total : 4987 time to create 1 rle with old method : 0.00579380989074707 time for calcul the mask position with numpy : 0.03343915939331055 nb_pixel_total : 5057 time to create 1 rle with old method : 0.005590915679931641 time for calcul the mask position with numpy : 0.034070730209350586 nb_pixel_total : 40460 time to create 1 rle with old method : 0.0438387393951416 time for calcul the mask position with numpy : 0.03861188888549805 nb_pixel_total : 8490 time to create 1 rle with old method : 0.009968280792236328 time for calcul the mask position with numpy : 0.035407066345214844 nb_pixel_total : 81950 time to create 1 rle with old method : 0.09061264991760254 create new chi : 2.2201831340789795 time to delete rle : 0.0008177757263183594 batch 1 Loaded 12 chid ids of type : 3726 Number RLEs to save : 7811 TO DO : save crop sub photo not yet done ! save time : 3.7096030712127686 nb_obj : 6 nb_hashtags : 2 time to prepare the origin masks : 3.8810665607452393 time for calcul the mask position with numpy : 0.3279104232788086 nb_pixel_total : 3302287 time to create 1 rle with new method : 0.8821878433227539 time for calcul the mask position with numpy : 0.02840447425842285 nb_pixel_total : 472978 time to create 1 rle with new method : 0.9085245132446289 time for calcul the mask position with numpy : 0.038227081298828125 nb_pixel_total : 349543 time to create 1 rle with new method : 0.6520287990570068 time for calcul the mask position with numpy : 0.03697514533996582 nb_pixel_total : 1999 time to create 1 rle with old method : 0.002244234085083008 time for calcul the mask position with numpy : 0.29854607582092285 nb_pixel_total : 2826148 time to create 1 rle with new method : 0.9148232936859131 time for calcul the mask position with numpy : 0.02105402946472168 nb_pixel_total : 9065 time to create 1 rle with old method : 0.009480476379394531 time for calcul the mask position with numpy : 0.021045923233032227 nb_pixel_total : 88220 time to create 1 rle with old method : 0.09810614585876465 create new chi : 4.358628988265991 time to delete rle : 0.0019562244415283203 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 15429 TO DO : save crop sub photo not yet done ! save time : 9.41701865196228 nb_obj : 14 nb_hashtags : 5 time to prepare the origin masks : 9.879860877990723 time for calcul the mask position with numpy : 0.27514076232910156 nb_pixel_total : 4341081 time to create 1 rle with new method : 1.1744592189788818 time for calcul the mask position with numpy : 0.02943873405456543 nb_pixel_total : 68261 time to create 1 rle with old method : 0.10653829574584961 time for calcul the mask position with numpy : 0.04202413558959961 nb_pixel_total : 54083 time to create 1 rle with old method : 0.06315398216247559 time for calcul the mask position with numpy : 0.043764352798461914 nb_pixel_total : 36854 time to create 1 rle with old method : 0.05089402198791504 time for calcul the mask position with numpy : 0.04728364944458008 nb_pixel_total : 138610 time to create 1 rle with old method : 0.18515276908874512 time for calcul the mask position with numpy : 0.03527975082397461 nb_pixel_total : 3262 time to create 1 rle with old method : 0.003808736801147461 time for calcul the mask position with numpy : 0.03815150260925293 nb_pixel_total : 724764 time to create 1 rle with new method : 0.9877195358276367 time for calcul the mask position with numpy : 0.024044036865234375 nb_pixel_total : 2561 time to create 1 rle with old method : 0.003154754638671875 time for calcul the mask position with numpy : 0.024388790130615234 nb_pixel_total : 14652 time to create 1 rle with old method : 0.016069650650024414 time for calcul the mask position with numpy : 0.035492897033691406 nb_pixel_total : 1207898 time to create 1 rle with new method : 1.118055820465088 time for calcul the mask position with numpy : 0.02277374267578125 nb_pixel_total : 83711 time to create 1 rle with old method : 0.09349393844604492 time for calcul the mask position with numpy : 0.024201393127441406 nb_pixel_total : 91662 time to create 1 rle with old method : 0.10298347473144531 time for calcul the mask position with numpy : 0.027901649475097656 nb_pixel_total : 61946 time to create 1 rle with old method : 0.07044172286987305 time for calcul the mask position with numpy : 0.03725099563598633 nb_pixel_total : 201212 time to create 1 rle with new method : 0.8679947853088379 time for calcul the mask position with numpy : 0.0318148136138916 nb_pixel_total : 19683 time to create 1 rle with old method : 0.022915363311767578 create new chi : 5.728283405303955 time to delete rle : 0.002537250518798828 batch 1 Loaded 15 chid ids of type : 3726 Number RLEs to save : 19339 TO DO : save crop sub photo not yet done ! save time : 8.613901615142822 nb_obj : 8 nb_hashtags : 6 time to prepare the origin masks : 5.107356548309326 time for calcul the mask position with numpy : 0.7231166362762451 nb_pixel_total : 6489395 time to create 1 rle with new method : 1.0079412460327148 time for calcul the mask position with numpy : 0.02249312400817871 nb_pixel_total : 95295 time to create 1 rle with old method : 0.10592293739318848 time for calcul the mask position with numpy : 0.03664803504943848 nb_pixel_total : 117784 time to create 1 rle with old method : 0.13640475273132324 time for calcul the mask position with numpy : 0.036455631256103516 nb_pixel_total : 216982 time to create 1 rle with new method : 0.8871190547943115 time for calcul the mask position with numpy : 0.03484010696411133 nb_pixel_total : 14089 time to create 1 rle with old method : 0.015615463256835938 time for calcul the mask position with numpy : 0.0347745418548584 nb_pixel_total : 79669 time to create 1 rle with old method : 0.08888936042785645 time for calcul the mask position with numpy : 0.033617258071899414 nb_pixel_total : 18640 time to create 1 rle with old method : 0.020121335983276367 time for calcul the mask position with numpy : 0.021643400192260742 nb_pixel_total : 5607 time to create 1 rle with old method : 0.00600743293762207 time for calcul the mask position with numpy : 0.02036905288696289 nb_pixel_total : 12779 time to create 1 rle with old method : 0.01390838623046875 create new chi : 3.307682752609253 time to delete rle : 0.0009469985961914062 batch 1 Loaded 9 chid ids of type : 3726 Number RLEs to save : 8738 TO DO : save crop sub photo not yet done ! save time : 2.0480284690856934 nb_obj : 18 nb_hashtags : 6 time to prepare the origin masks : 16.006922245025635 time for calcul the mask position with numpy : 0.4331626892089844 nb_pixel_total : 4041289 time to create 1 rle with new method : 0.8508334159851074 time for calcul the mask position with numpy : 0.028688669204711914 nb_pixel_total : 307060 time to create 1 rle with new method : 0.7390947341918945 time for calcul the mask position with numpy : 0.03756546974182129 nb_pixel_total : 1223 time to create 1 rle with old method : 0.0015473365783691406 time for calcul the mask position with numpy : 0.03606128692626953 nb_pixel_total : 9668 time to create 1 rle with old method : 0.01153254508972168 time for calcul the mask position with numpy : 0.03592872619628906 nb_pixel_total : 5815 time to create 1 rle with old method : 0.006539583206176758 time for calcul the mask position with numpy : 0.0381011962890625 nb_pixel_total : 55614 time to create 1 rle with old method : 0.0620272159576416 time for calcul the mask position with numpy : 0.0508880615234375 nb_pixel_total : 37995 time to create 1 rle with old method : 0.043648719787597656 time for calcul the mask position with numpy : 0.046184539794921875 nb_pixel_total : 157993 time to create 1 rle with new method : 1.1204195022583008 time for calcul the mask position with numpy : 0.03510689735412598 nb_pixel_total : 65408 time to create 1 rle with old method : 0.07562613487243652 time for calcul the mask position with numpy : 0.052483558654785156 nb_pixel_total : 731815 time to create 1 rle with new method : 0.9919981956481934 time for calcul the mask position with numpy : 0.03058004379272461 nb_pixel_total : 10701 time to create 1 rle with old method : 0.05556750297546387 time for calcul the mask position with numpy : 0.0284273624420166 nb_pixel_total : 30381 time to create 1 rle with old method : 0.03383231163024902 time for calcul the mask position with numpy : 0.023763418197631836 nb_pixel_total : 123347 time to create 1 rle with old method : 0.13930702209472656 time for calcul the mask position with numpy : 0.02417778968811035 nb_pixel_total : 16221 time to create 1 rle with old method : 0.02717137336730957 time for calcul the mask position with numpy : 0.05162239074707031 nb_pixel_total : 1132054 time to create 1 rle with new method : 0.9128522872924805 time for calcul the mask position with numpy : 0.043025970458984375 nb_pixel_total : 95439 time to create 1 rle with old method : 0.10897135734558105 time for calcul the mask position with numpy : 0.0418243408203125 nb_pixel_total : 20490 time to create 1 rle with old method : 0.022244691848754883 time for calcul the mask position with numpy : 0.04174661636352539 nb_pixel_total : 102274 time to create 1 rle with old method : 0.11617207527160645 time for calcul the mask position with numpy : 0.042405128479003906 nb_pixel_total : 105453 time to create 1 rle with old method : 0.11597394943237305 create new chi : 6.6929755210876465 time to delete rle : 0.002187967300415039 batch 1 Loaded 19 chid ids of type : 3726 Number RLEs to save : 22597 TO DO : save crop sub photo not yet done ! save time : 4.8322906494140625 map_output_result : {1340898539: (0.0, 'Should be the crop_list due to order', 0), 1340898531: (0.0, 'Should be the crop_list due to order', 0), 1340898528: (0.0, 'Should be the crop_list due to order', 0), 1340898525: (0.0, 'Should be the crop_list due to order', 0), 1340898409: (0.0, 'Should be the crop_list due to order', 0), 1340898402: (0.0, 'Should be the crop_list due to order', 0), 1340898397: (0.0, 'Should be the crop_list due to order', 0), 1340898304: (0.0, 'Should be the crop_list due to order', 0), 1340898086: (0.0, 'Should be the crop_list due to order', 0), 1340898002: (0.0, 'Should be the crop_list due to order', 0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1340898539, 1340898531, 1340898528, 1340898525, 1340898409, 1340898402, 1340898397, 1340898304, 1340898086, 1340898002] Looping around the photos to save general results len do output : 10 /1340898539.Didn't retrieve data . /1340898531.Didn't retrieve data . /1340898528.Didn't retrieve data . /1340898525.Didn't retrieve data . /1340898409.Didn't retrieve data . /1340898402.Didn't retrieve data . /1340898397.Didn't retrieve data . /1340898304.Didn't retrieve data . /1340898086.Didn't retrieve data . /1340898002.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898539', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898531', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898528', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898525', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898409', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898402', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898397', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898304', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898086', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898002', None, None, None, None, None, '2635391') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.019850730895996094 save_final save missing photos in datou_result : time spend for datou_step_exec : 192.44488215446472 time spend to save output : 0.0240023136138916 total time spend for step 5 : 192.4688844680786 step6:crop_condition Wed Mar 5 04:16:16 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 146 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741144581_3182781 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.866642951965332 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741144586_3182781 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.1825077533721924 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! About to upload 8 photos upload in portfolio : 4869462 init cache_photo without model_param we have 8 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741144594_3182781 we have uploaded 8 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.433629035949707 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741144599_3182781 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.3316402435302734 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741144602_3182781 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.790311574935913 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741144607_3182781 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.9609837532043457 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741144610_3182781 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.7088942527770996 we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1340898539, 1340898531, 1340898528, 1340898525, 1340898409, 1340898402, 1340898397, 1340898304, 1340898086, 1340898002] Looping around the photos to save general results len do output : 31 /1340928825Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928826Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928827Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928828Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928829Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928830Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928832Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928833Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928834Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928844Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928845Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928846Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928847Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928848Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928850Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928851Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928852Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928856Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928857Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928858Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928859Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928864Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928865Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928866Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928867Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928868Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928869Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928978Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928979Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928980Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340928982Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898539', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898531', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898528', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898525', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898409', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898402', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898397', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898304', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898086', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898002', None, None, None, None, None, '2635391') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 103 time used for this insertion : 0.03617691993713379 save_final save missing photos in datou_result : time spend for datou_step_exec : 36.56664776802063 time spend to save output : 0.03937840461730957 total time spend for step 6 : 36.60602617263794 step7:ventilate_hashtags_in_portfolio Wed Mar 5 04:16:53 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 21042321 get user id for portfolio 21042321 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21042321 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('flou','papier','metal','mal_croppe','Carton_brun','Teint_Dans_La_Masse','environnement','kraft','cartonnette','plastique','autre_refus','Carton_gris')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21042321 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('flou','papier','metal','mal_croppe','Carton_brun','Teint_Dans_La_Masse','environnement','kraft','cartonnette','plastique','autre_refus','Carton_gris')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21042321 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('flou','papier','metal','mal_croppe','Carton_brun','Teint_Dans_La_Masse','environnement','kraft','cartonnette','plastique','autre_refus','Carton_gris')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/21042503,21042504,21042505,21042506,21042507,21042508,21042509,21042510,21042511,21042512,21042513,21042514?tags=flou,papier,metal,mal_croppe,Carton_brun,Teint_Dans_La_Masse,environnement,kraft,cartonnette,plastique,autre_refus,Carton_gris Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1340898539, 1340898531, 1340898528, 1340898525, 1340898409, 1340898402, 1340898397, 1340898304, 1340898086, 1340898002] Looping around the photos to save general results len do output : 1 /21042321. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898539', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898531', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898528', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898525', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898409', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898402', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898397', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898304', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898086', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898002', None, None, None, None, None, '2635391') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.028467416763305664 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.846632719039917 time spend to save output : 0.028717041015625 total time spend for step 7 : 1.875349760055542 step8:final Wed Mar 5 04:16:55 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1340898539: ('0.09847279539732458',), 1340898531: ('0.09847279539732458',), 1340898528: ('0.09847279539732458',), 1340898525: ('0.09847279539732458',), 1340898409: ('0.09847279539732458',), 1340898402: ('0.09847279539732458',), 1340898397: ('0.09847279539732458',), 1340898304: ('0.09847279539732458',), 1340898086: ('0.09847279539732458',), 1340898002: ('0.09847279539732458',)} new output for save of step final : {1340898539: ('0.09847279539732458',), 1340898531: ('0.09847279539732458',), 1340898528: ('0.09847279539732458',), 1340898525: ('0.09847279539732458',), 1340898409: ('0.09847279539732458',), 1340898402: ('0.09847279539732458',), 1340898397: ('0.09847279539732458',), 1340898304: ('0.09847279539732458',), 1340898086: ('0.09847279539732458',), 1340898002: ('0.09847279539732458',)} [1340898539, 1340898531, 1340898528, 1340898525, 1340898409, 1340898402, 1340898397, 1340898304, 1340898086, 1340898002] Looping around the photos to save general results len do output : 10 /1340898539.Didn't retrieve data . /1340898531.Didn't retrieve data . /1340898528.Didn't retrieve data . /1340898525.Didn't retrieve data . /1340898409.Didn't retrieve data . /1340898402.Didn't retrieve data . /1340898397.Didn't retrieve data . /1340898304.Didn't retrieve data . /1340898086.Didn't retrieve data . /1340898002.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898539', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898531', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898528', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898525', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898409', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898402', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898397', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898304', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898086', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898002', None, None, None, None, None, '2635391') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.016422748565673828 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.11145710945129395 time spend to save output : 0.016995668411254883 total time spend for step 8 : 0.12845277786254883 step9:velours_tree Wed Mar 5 04:16:55 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.11773324012756348 time spend to save output : 4.1484832763671875e-05 total time spend for step 9 : 0.11777472496032715 step10:send_mail_cod Wed Mar 5 04:16:55 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P21042321_05-03-2025_04_16_55.pdf 21042503 imagette210425031741144615 21042505 imagette210425051741144615 21042506 imagette210425061741144615 21042510 change filename to text .imagette210425101741144615 21042511 change filename to text .change filename to text .change filename to text .change filename to text .imagette210425111741144615 21042512 change filename to text .change filename to text .change filename to text .imagette210425121741144615 21042513 change filename to text .change filename to text .change filename to text .imagette210425131741144615 21042507 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette210425071741144616 21042508 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette210425081741144616 21042514 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette210425141741144616 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=21042321 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/21042503,21042504,21042505,21042506,21042507,21042508,21042509,21042510,21042511,21042512,21042513,21042514?tags=flou,papier,metal,mal_croppe,Carton_brun,Teint_Dans_La_Masse,environnement,kraft,cartonnette,plastique,autre_refus,Carton_gris your option no_mail is active, we will not send the real mail to your client args[1340898539] : ('0.09847279539732458',) no score found for photo 1340898539 We are sending mail with results at report@fotonower.com args[1340898531] : ('0.09847279539732458',) no score found for photo 1340898531 We are sending mail with results at report@fotonower.com args[1340898528] : ('0.09847279539732458',) no score found for photo 1340898528 We are sending mail with results at report@fotonower.com args[1340898525] : ('0.09847279539732458',) no score found for photo 1340898525 We are sending mail with results at report@fotonower.com args[1340898409] : ('0.09847279539732458',) no score found for photo 1340898409 We are sending mail with results at report@fotonower.com args[1340898402] : ('0.09847279539732458',) no score found for photo 1340898402 We are sending mail with results at report@fotonower.com args[1340898397] : ('0.09847279539732458',) no score found for photo 1340898397 We are sending mail with results at report@fotonower.com args[1340898304] : ('0.09847279539732458',) no score found for photo 1340898304 We are sending mail with results at report@fotonower.com args[1340898086] : ('0.09847279539732458',) no score found for photo 1340898086 We are sending mail with results at report@fotonower.com args[1340898002] : ('0.09847279539732458',) no score found for photo 1340898002 We are sending mail with results at report@fotonower.com refus_total : 0.09847279539732458 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=21042321 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1340898086,1340898409,1340898525,1340898528,1340898531,1340898539,1340898002,1340898304,1340898397,1340898402) Found this number of photos: 10 begin to download photo : 1340898086 begin to download photo : 1340898528 begin to download photo : 1340898002 begin to download photo : 1340898402 download finish for photo 1340898086 begin to download photo : 1340898409 download finish for photo 1340898002 begin to download photo : 1340898304 download finish for photo 1340898402 download finish for photo 1340898528 begin to download photo : 1340898531 download finish for photo 1340898409 begin to download photo : 1340898525 download finish for photo 1340898304 begin to download photo : 1340898397 download finish for photo 1340898397 download finish for photo 1340898531 begin to download photo : 1340898539 download finish for photo 1340898525 download finish for photo 1340898539 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21042321_05-03-2025_04_16_55.pdf results_Auto_P21042321_05-03-2025_04_16_55.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21042321_05-03-2025_04_16_55.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','21042321','results_Auto_P21042321_05-03-2025_04_16_55.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21042321_05-03-2025_04_16_55.pdf','pdf','','0.58','0.09847279539732458') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1340898539, 1340898531, 1340898528, 1340898525, 1340898409, 1340898402, 1340898397, 1340898304, 1340898086, 1340898002] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898539', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898531', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898528', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898525', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898409', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898402', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898397', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898304', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898086', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898002', None, None, None, None, None, '2635391') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.01382589340209961 save_final save missing photos in datou_result : time spend for datou_step_exec : 4.794902563095093 time spend to save output : 0.014039993286132812 total time spend for step 10 : 4.808942556381226 step11:split_time_score Wed Mar 5 04:17:00 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('09', 10),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 04032025 21042321 Nombre de photos uploadées : 10 / 23040 (0%) 04032025 21042321 Nombre de photos taguées (types de déchets): 0 / 10 (0%) 04032025 21042321 Nombre de photos taguées (volume) : 0 / 10 (0%) elapsed_time : load_data_split_time_score 1.9073486328125e-06 elapsed_time : order_list_meta_photo_and_scores 5.7220458984375e-06 ?????????? elapsed_time : fill_and_build_computed_from_old_data 0.0007317066192626953 elapsed_time : insert_dashboard_record_day_entry 0.029727697372436523 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21033665 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21042319 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21033667 order by id desc limit 1 Qualite : 0.06655189856603971 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21033668_05-03-2025_02_13_19.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21033668 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21033668 AND mptpi.`type`=4207 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21042320 order by id desc limit 1 Catched exception ! Connect or reconnect ! Qualite : 0.09847279539732458 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21042321_05-03-2025_04_16_55.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21042321 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21042321 AND mptpi.`type`=3726 To do Qualite : 0.06750380359255952 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21035844_05-03-2025_01_47_30.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21035844 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11524 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11533 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11532 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11528 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11528 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11531 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11531 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11525 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11525 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11578 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11578 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11527 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11526 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11526 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11535 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11527 doesn't seem to be define in the database( WARNING : type of input 3 of step 11526 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11524 doesn't seem to be define in the database( WARNING : type of input 2 of step 11528 doesn't seem to be define in the database( WARNING : output 1 of step 11524 have datatype=2 whereas input 1 of step 11531 have datatype=7 WARNING : type of output 2 of step 11531 doesn't seem to be define in the database( WARNING : type of input 1 of step 11525 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11527 have datatype=10 whereas input 3 of step 11534 have datatype=6 WARNING : type of input 2 of step 11578 doesn't seem to be define in the database( WARNING : output 1 of step 11525 have datatype=7 whereas input 2 of step 11578 have datatype=None WARNING : type of output 3 of step 11578 doesn't seem to be define in the database( WARNING : type of input 1 of step 11527 doesn't seem to be define in the database( WARNING : output 0 of step 11527 have datatype=10 whereas input 0 of step 11584 have datatype=18 WARNING : type of input 5 of step 11534 doesn't seem to be define in the database( WARNING : output 0 of step 11584 have datatype=11 whereas input 5 of step 11534 have datatype=None WARNING : type of output 1 of step 11533 doesn't seem to be define in the database( WARNING : type of input 3 of step 11528 doesn't seem to be define in the database( WARNING : type of output 1 of step 11532 doesn't seem to be define in the database( WARNING : type of input 3 of step 11528 doesn't seem to be define in the database( WARNING : output 0 of step 11531 have datatype=1 whereas input 0 of step 11525 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21035844 AND mptpi.`type`=4211 To do Qualite : 0.09362080469662035 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21040185_05-03-2025_04_09_01.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21040185 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21040185 AND mptpi.`type`=4207 To do Qualite : 0.07900093100209174 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21040186_05-03-2025_03_55_22.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21040186 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21040186 AND mptpi.`type`=4207 To do Qualite : 0.061101602071808085 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21040187_05-03-2025_03_55_09.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21040187 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21040187 AND mptpi.`type`=4207 To do Qualite : 0.06662977779146534 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21037399_05-03-2025_03_06_36.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21037399 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21037399 AND mptpi.`type`=4207 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21037400 order by id desc limit 1 Qualite : 0.48436678153156526 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21037401_05-03-2025_04_16_08.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21037401 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11548 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11556 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11557 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11552 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11552 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11555 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11555 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11580 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11580 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11551 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11550 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11550 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11559 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11551 doesn't seem to be define in the database( WARNING : type of input 3 of step 11550 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11548 doesn't seem to be define in the database( WARNING : type of input 2 of step 11552 doesn't seem to be define in the database( WARNING : output 1 of step 11548 have datatype=2 whereas input 1 of step 11555 have datatype=7 WARNING : type of output 2 of step 11555 doesn't seem to be define in the database( WARNING : type of input 1 of step 11549 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11551 have datatype=10 whereas input 3 of step 11558 have datatype=6 WARNING : type of input 2 of step 11580 doesn't seem to be define in the database( WARNING : output 1 of step 11549 have datatype=7 whereas input 2 of step 11580 have datatype=None WARNING : type of output 3 of step 11580 doesn't seem to be define in the database( WARNING : type of input 1 of step 11551 doesn't seem to be define in the database( WARNING : output 0 of step 11551 have datatype=10 whereas input 0 of step 11586 have datatype=18 WARNING : type of input 5 of step 11558 doesn't seem to be define in the database( WARNING : output 0 of step 11586 have datatype=11 whereas input 5 of step 11558 have datatype=None WARNING : type of output 1 of step 11556 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : type of output 1 of step 11557 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : output 0 of step 11555 have datatype=1 whereas input 0 of step 11549 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21037401 AND mptpi.`type`=4200 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21037402 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21037403 order by id desc limit 1 Qualite : 0.10012831451353366 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21040188_05-03-2025_03_37_19.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21040188 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21040188 AND mptpi.`type`=4207 To do Qualite : 0.06878107338417522 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21035849_05-03-2025_01_42_54.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21035849 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21035849 AND mptpi.`type`=4207 To do Qualite : 0.044771638969161004 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21037404_05-03-2025_02_29_47.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21037404 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21037404 AND mptpi.`type`=4207 To do Qualite : 0.09177931191826523 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21040189_05-03-2025_03_40_10.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21040189 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21040189 AND mptpi.`type`=4207 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21042322 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21042323 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21042324 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21033673 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21033674 order by id desc limit 1 Qualite : 0.22923686414801614 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21033675_05-03-2025_00_21_09.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21033675 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21033675 AND mptpi.`type`=3594 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21033676 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21033677 order by id desc limit 1 Qualite : 0.10784020504319093 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21033678_05-03-2025_03_10_04.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21033678 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21033678 AND mptpi.`type`=4207 To do Qualite : 0.0786377704762859 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21033679_05-03-2025_00_42_03.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21033679 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21033679 AND mptpi.`type`=4207 To do Qualite : 0.05450229800969905 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21033680_05-03-2025_00_26_20.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21033680 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21033680 AND mptpi.`type`=4207 To do Qualite : 0.09179124235004173 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21033681_05-03-2025_03_18_30.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21033681 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21033681 AND mptpi.`type`=3726 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'04032025': {'nb_upload': 10, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1340898539, 1340898531, 1340898528, 1340898525, 1340898409, 1340898402, 1340898397, 1340898304, 1340898086, 1340898002] Looping around the photos to save general results len do output : 1 /21042321Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898539', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898531', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898528', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898525', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898409', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898402', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898397', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898304', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898086', None, None, None, None, None, '2635391') ('3459', None, None, None, None, None, None, None, '2635391') ('3459', '21042321', '1340898002', None, None, None, None, None, '2635391') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.017231464385986328 save_final save missing photos in datou_result : time spend for datou_step_exec : 12.6913480758667 time spend to save output : 0.01745009422302246 total time spend for step 11 : 12.708798170089722 caffe_path_current : About to save ! 2 After save, about to update current ! ret : 1 len(input) + len(total_photo_id_missing) : 10 set_done_treatment 162.40user 165.05system 6:49.45elapsed 79%CPU (0avgtext+0avgdata 3958228maxresident)k 1225880inputs+110352outputs (18455major+14288934minor)pagefaults 0swaps