python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 2101163 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 37>10 please execute split_portfolio.py -i 20321739 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2561530'] with mtr_portfolio_ids : ['20321739'] and first list_photo_ids : [] new path : /proc/2101163/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 2.491527795791626 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Fri Feb 7 06:38:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 3000 l 3632 free memory gpu now : 159 wait 20 seconds l 3637 free memory gpu now : 159 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-07 06:38:52.502289: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-07 06:38:52.527125: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-07 06:38:52.528944: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fbe80000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-07 06:38:52.528973: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-07 06:38:52.532635: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-07 06:38:52.780701: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x249b6340 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-07 06:38:52.780739: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-07 06:38:52.781659: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-07 06:38:52.781962: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-07 06:38:52.784263: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-07 06:38:52.786386: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-07 06:38:52.786740: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-07 06:38:52.794182: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-07 06:38:52.795570: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-07 06:38:52.802636: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-07 06:38:52.804064: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-07 06:38:52.804161: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-07 06:38:52.804890: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-07 06:38:52.804908: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-07 06:38:52.804918: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-07 06:38:52.806187: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4234 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-07 06:38:53.082095: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-07 06:38:53.082188: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-07 06:38:53.082212: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-07 06:38:53.082232: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-07 06:38:53.082251: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-07 06:38:53.082271: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-07 06:38:53.082290: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-07 06:38:53.082310: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-07 06:38:53.083519: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-07 06:38:53.084553: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-07 06:38:53.084591: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-07 06:38:53.084613: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-07 06:38:53.084636: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-07 06:38:53.084655: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-07 06:38:53.084675: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-07 06:38:53.084695: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-07 06:38:53.084714: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-07 06:38:53.085902: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-07 06:38:53.085934: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-07 06:38:53.085946: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-07 06:38:53.085957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-07 06:38:53.087260: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4234 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-07 06:39:04.449815: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-07 06:39:04.624761: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-07 06:39:05.862208: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.64G (3904765952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-07 06:39:06.700084: W tensorflow/core/common_runtime/bfc_allocator.cc:311] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature. 2025-02-07 06:39:06.781236: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.64G (3904765952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-07 06:39:06.781321: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-07 06:39:06.782077: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.64G (3904765952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-07 06:39:06.782096: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-07 06:39:06.788510: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.64G (3904765952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-07 06:39:06.788532: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.78GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-07 06:39:06.789222: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.64G (3904765952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-07 06:39:06.789236: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.78GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-07 06:39:06.846071: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.64G (3904765952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-07 06:39:06.846115: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.26GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-07 06:39:06.846875: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.64G (3904765952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-07 06:39:06.846893: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.26GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-07 06:39:06.854892: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.64G (3904765952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-07 06:39:06.855141: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 417.62MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-07 06:39:06.856579: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.64G (3904765952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-07 06:39:06.856609: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 417.62MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-07 06:39:06.873047: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.64G (3904765952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-07 06:39:06.873085: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 188.27MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-07 06:39:06.874027: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.64G (3904765952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-07 06:39:06.874048: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 188.27MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-07 06:39:06.875093: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.64G (3904765952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-07 06:39:06.876043: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.64G (3904765952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-07 06:39:06.880670: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.64G (3904765952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-07 06:39:06.881424: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.64G (3904765952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-07 06:39:06.882181: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.64G (3904765952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-07 06:39:06.882917: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.64G (3904765952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-07 06:39:06.884061: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.64G (3904765952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-07 06:39:06.884078: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-02-07 06:39:06.894395: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.64G (3904765952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-07 06:39:06.895166: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.64G (3904765952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-07 06:39:06.904336: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.64G (3904765952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-07 06:39:06.905059: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.64G (3904765952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-07 06:39:06.905764: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.64G (3904765952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-07 06:39:06.906460: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.64G (3904765952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-07 06:39:06.907443: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.64G (3904765952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-07 06:39:06.908453: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.64G (3904765952 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 20 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 23 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 14 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 2 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 9 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 22 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 29 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 25 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 14 Detection mask done ! Trying to reset tf kernel 2101846 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 3493 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 4686 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.0006628036499023438 nb_pixel_total : 33105 time to create 1 rle with old method : 0.03719806671142578 length of segment : 248 time for calcul the mask position with numpy : 0.00019311904907226562 nb_pixel_total : 5458 time to create 1 rle with old method : 0.006776094436645508 length of segment : 106 time for calcul the mask position with numpy : 8.463859558105469e-05 nb_pixel_total : 3200 time to create 1 rle with old method : 0.003800630569458008 length of segment : 89 time for calcul the mask position with numpy : 0.0014607906341552734 nb_pixel_total : 34606 time to create 1 rle with old method : 0.03941082954406738 length of segment : 484 time for calcul the mask position with numpy : 0.00015473365783691406 nb_pixel_total : 5364 time to create 1 rle with old method : 0.006256818771362305 length of segment : 154 time for calcul the mask position with numpy : 0.001039743423461914 nb_pixel_total : 58395 time to create 1 rle with old method : 0.06517863273620605 length of segment : 330 time for calcul the mask position with numpy : 0.0008220672607421875 nb_pixel_total : 58947 time to create 1 rle with old method : 0.06382608413696289 length of segment : 265 time for calcul the mask position with numpy : 0.00018286705017089844 nb_pixel_total : 7746 time to create 1 rle with old method : 0.00931549072265625 length of segment : 111 time for calcul the mask position with numpy : 0.0029468536376953125 nb_pixel_total : 229432 time to create 1 rle with new method : 0.008594512939453125 length of segment : 513 time for calcul the mask position with numpy : 0.00021696090698242188 nb_pixel_total : 9979 time to create 1 rle with old method : 0.011092662811279297 length of segment : 152 time for calcul the mask position with numpy : 0.0006415843963623047 nb_pixel_total : 43700 time to create 1 rle with old method : 0.046805620193481445 length of segment : 270 time for calcul the mask position with numpy : 6.67572021484375e-05 nb_pixel_total : 1380 time to create 1 rle with old method : 0.0016813278198242188 length of segment : 75 time for calcul the mask position with numpy : 0.0005824565887451172 nb_pixel_total : 39068 time to create 1 rle with old method : 0.04325103759765625 length of segment : 327 time for calcul the mask position with numpy : 0.001661539077758789 nb_pixel_total : 40439 time to create 1 rle with old method : 0.06282877922058105 length of segment : 492 time for calcul the mask position with numpy : 0.00031638145446777344 nb_pixel_total : 17145 time to create 1 rle with old method : 0.019280433654785156 length of segment : 208 time for calcul the mask position with numpy : 0.0044748783111572266 nb_pixel_total : 330325 time to create 1 rle with new method : 0.015179872512817383 length of segment : 880 time for calcul the mask position with numpy : 0.0003459453582763672 nb_pixel_total : 17805 time to create 1 rle with old method : 0.02055191993713379 length of segment : 168 time for calcul the mask position with numpy : 7.319450378417969e-05 nb_pixel_total : 1865 time to create 1 rle with old method : 0.002384185791015625 length of segment : 50 time for calcul the mask position with numpy : 0.001924753189086914 nb_pixel_total : 127534 time to create 1 rle with old method : 0.14094090461730957 length of segment : 512 time for calcul the mask position with numpy : 0.0007412433624267578 nb_pixel_total : 50423 time to create 1 rle with old method : 0.0560154914855957 length of segment : 219 time for calcul the mask position with numpy : 9.846687316894531e-05 nb_pixel_total : 3559 time to create 1 rle with old method : 0.004343748092651367 length of segment : 95 time for calcul the mask position with numpy : 0.001367330551147461 nb_pixel_total : 96421 time to create 1 rle with old method : 0.10715842247009277 length of segment : 333 time for calcul the mask position with numpy : 0.00011491775512695312 nb_pixel_total : 5374 time to create 1 rle with old method : 0.0059871673583984375 length of segment : 67 time for calcul the mask position with numpy : 0.0001556873321533203 nb_pixel_total : 2641 time to create 1 rle with old method : 0.003187894821166992 length of segment : 90 time for calcul the mask position with numpy : 0.0014047622680664062 nb_pixel_total : 77891 time to create 1 rle with old method : 0.08480000495910645 length of segment : 260 time for calcul the mask position with numpy : 0.0020186901092529297 nb_pixel_total : 108880 time to create 1 rle with old method : 0.12976980209350586 length of segment : 392 time for calcul the mask position with numpy : 0.0008351802825927734 nb_pixel_total : 40358 time to create 1 rle with old method : 0.046941518783569336 length of segment : 405 time for calcul the mask position with numpy : 0.0002701282501220703 nb_pixel_total : 12957 time to create 1 rle with old method : 0.016256093978881836 length of segment : 137 time for calcul the mask position with numpy : 0.0001983642578125 nb_pixel_total : 13162 time to create 1 rle with old method : 0.01543283462524414 length of segment : 92 time for calcul the mask position with numpy : 0.0010676383972167969 nb_pixel_total : 71343 time to create 1 rle with old method : 0.08060169219970703 length of segment : 232 time for calcul the mask position with numpy : 0.0020270347595214844 nb_pixel_total : 109628 time to create 1 rle with old method : 0.12272763252258301 length of segment : 504 time for calcul the mask position with numpy : 0.00032973289489746094 nb_pixel_total : 14718 time to create 1 rle with old method : 0.01678013801574707 length of segment : 139 time for calcul the mask position with numpy : 0.00010943412780761719 nb_pixel_total : 2510 time to create 1 rle with old method : 0.0028319358825683594 length of segment : 105 time for calcul the mask position with numpy : 8.559226989746094e-05 nb_pixel_total : 2588 time to create 1 rle with old method : 0.0031108856201171875 length of segment : 84 time for calcul the mask position with numpy : 0.00133514404296875 nb_pixel_total : 71850 time to create 1 rle with old method : 0.08094263076782227 length of segment : 254 time for calcul the mask position with numpy : 0.000324249267578125 nb_pixel_total : 15370 time to create 1 rle with old method : 0.01741647720336914 length of segment : 141 time for calcul the mask position with numpy : 0.0007269382476806641 nb_pixel_total : 41281 time to create 1 rle with old method : 0.044946908950805664 length of segment : 341 time for calcul the mask position with numpy : 7.987022399902344e-05 nb_pixel_total : 1712 time to create 1 rle with old method : 0.002130270004272461 length of segment : 61 time for calcul the mask position with numpy : 0.00011348724365234375 nb_pixel_total : 4459 time to create 1 rle with old method : 0.005151033401489258 length of segment : 102 time for calcul the mask position with numpy : 0.7034578323364258 nb_pixel_total : 1523893 time to create 1 rle with new method : 0.7729752063751221 length of segment : 1796 time for calcul the mask position with numpy : 0.0006985664367675781 nb_pixel_total : 39245 time to create 1 rle with old method : 0.04245638847351074 length of segment : 348 time for calcul the mask position with numpy : 0.0006575584411621094 nb_pixel_total : 53534 time to create 1 rle with old method : 0.056311845779418945 length of segment : 233 time for calcul the mask position with numpy : 0.0032854080200195312 nb_pixel_total : 294340 time to create 1 rle with new method : 0.007626533508300781 length of segment : 687 time for calcul the mask position with numpy : 0.0009033679962158203 nb_pixel_total : 46077 time to create 1 rle with old method : 0.05229973793029785 length of segment : 340 time for calcul the mask position with numpy : 0.0007688999176025391 nb_pixel_total : 49893 time to create 1 rle with old method : 0.0537569522857666 length of segment : 353 time for calcul the mask position with numpy : 7.867813110351562e-05 nb_pixel_total : 1531 time to create 1 rle with old method : 0.0020067691802978516 length of segment : 56 time for calcul the mask position with numpy : 0.0028541088104248047 nb_pixel_total : 178464 time to create 1 rle with new method : 0.007318735122680664 length of segment : 456 time for calcul the mask position with numpy : 0.0016543865203857422 nb_pixel_total : 95610 time to create 1 rle with old method : 0.10197186470031738 length of segment : 550 time for calcul the mask position with numpy : 0.0004322528839111328 nb_pixel_total : 26951 time to create 1 rle with old method : 0.029174327850341797 length of segment : 183 time for calcul the mask position with numpy : 0.00034618377685546875 nb_pixel_total : 25901 time to create 1 rle with old method : 0.02851271629333496 length of segment : 188 time for calcul the mask position with numpy : 0.31111931800842285 nb_pixel_total : 3295776 time to create 1 rle with new method : 1.260239839553833 length of segment : 3429 time for calcul the mask position with numpy : 0.0004780292510986328 nb_pixel_total : 9326 time to create 1 rle with old method : 0.011554718017578125 length of segment : 121 time for calcul the mask position with numpy : 0.355820894241333 nb_pixel_total : 3393589 time to create 1 rle with new method : 0.7573530673980713 length of segment : 3103 time for calcul the mask position with numpy : 0.0006964206695556641 nb_pixel_total : 31537 time to create 1 rle with old method : 0.034745216369628906 length of segment : 317 time for calcul the mask position with numpy : 0.0010523796081542969 nb_pixel_total : 55071 time to create 1 rle with old method : 0.0620574951171875 length of segment : 316 time for calcul the mask position with numpy : 0.0008070468902587891 nb_pixel_total : 62773 time to create 1 rle with old method : 0.06894493103027344 length of segment : 281 time for calcul the mask position with numpy : 0.00492095947265625 nb_pixel_total : 372504 time to create 1 rle with new method : 0.010329008102416992 length of segment : 755 time for calcul the mask position with numpy : 0.0001697540283203125 nb_pixel_total : 3090 time to create 1 rle with old method : 0.003842592239379883 length of segment : 82 time for calcul the mask position with numpy : 0.0012400150299072266 nb_pixel_total : 56053 time to create 1 rle with old method : 0.06283426284790039 length of segment : 297 time for calcul the mask position with numpy : 0.3743412494659424 nb_pixel_total : 4279921 time to create 1 rle with new method : 0.6050255298614502 length of segment : 3238 time for calcul the mask position with numpy : 0.00018906593322753906 nb_pixel_total : 8442 time to create 1 rle with old method : 0.010164976119995117 length of segment : 69 time for calcul the mask position with numpy : 0.0009262561798095703 nb_pixel_total : 74830 time to create 1 rle with old method : 0.08324122428894043 length of segment : 375 time for calcul the mask position with numpy : 0.00087738037109375 nb_pixel_total : 65352 time to create 1 rle with old method : 0.07084774971008301 length of segment : 267 time for calcul the mask position with numpy : 0.0002522468566894531 nb_pixel_total : 13660 time to create 1 rle with old method : 0.015636205673217773 length of segment : 159 time for calcul the mask position with numpy : 0.0003256797790527344 nb_pixel_total : 8268 time to create 1 rle with old method : 0.009348630905151367 length of segment : 110 time for calcul the mask position with numpy : 0.0004928112030029297 nb_pixel_total : 7317 time to create 1 rle with old method : 0.008369922637939453 length of segment : 181 time for calcul the mask position with numpy : 0.005453824996948242 nb_pixel_total : 187450 time to create 1 rle with new method : 0.011031389236450195 length of segment : 978 time for calcul the mask position with numpy : 0.002200603485107422 nb_pixel_total : 68548 time to create 1 rle with old method : 0.08260869979858398 length of segment : 260 time for calcul the mask position with numpy : 0.0012536048889160156 nb_pixel_total : 35348 time to create 1 rle with old method : 0.038925886154174805 length of segment : 293 time for calcul the mask position with numpy : 0.0002651214599609375 nb_pixel_total : 5882 time to create 1 rle with old method : 0.006975412368774414 length of segment : 67 time for calcul the mask position with numpy : 0.0183868408203125 nb_pixel_total : 720074 time to create 1 rle with new method : 0.09695315361022949 length of segment : 1339 time for calcul the mask position with numpy : 0.00041604042053222656 nb_pixel_total : 11518 time to create 1 rle with old method : 0.012928247451782227 length of segment : 115 time for calcul the mask position with numpy : 0.04894733428955078 nb_pixel_total : 1420458 time to create 1 rle with new method : 0.4323582649230957 length of segment : 2163 time for calcul the mask position with numpy : 0.0014014244079589844 nb_pixel_total : 86815 time to create 1 rle with old method : 0.09342718124389648 length of segment : 398 time for calcul the mask position with numpy : 0.014606475830078125 nb_pixel_total : 520758 time to create 1 rle with new method : 0.059762001037597656 length of segment : 1252 time for calcul the mask position with numpy : 0.0005533695220947266 nb_pixel_total : 27481 time to create 1 rle with old method : 0.030176162719726562 length of segment : 139 time for calcul the mask position with numpy : 0.002842426300048828 nb_pixel_total : 150850 time to create 1 rle with new method : 0.008178949356079102 length of segment : 430 time for calcul the mask position with numpy : 7.653236389160156e-05 nb_pixel_total : 1256 time to create 1 rle with old method : 0.0016148090362548828 length of segment : 64 time for calcul the mask position with numpy : 0.00019121170043945312 nb_pixel_total : 9757 time to create 1 rle with old method : 0.011563301086425781 length of segment : 133 time for calcul the mask position with numpy : 0.0003173351287841797 nb_pixel_total : 20375 time to create 1 rle with old method : 0.022449970245361328 length of segment : 160 time for calcul the mask position with numpy : 0.0023238658905029297 nb_pixel_total : 156997 time to create 1 rle with new method : 0.006325960159301758 length of segment : 856 time for calcul the mask position with numpy : 0.003145933151245117 nb_pixel_total : 216344 time to create 1 rle with new method : 0.007573366165161133 length of segment : 626 time for calcul the mask position with numpy : 0.0004951953887939453 nb_pixel_total : 6050 time to create 1 rle with old method : 0.007340431213378906 length of segment : 118 time for calcul the mask position with numpy : 0.0015511512756347656 nb_pixel_total : 45474 time to create 1 rle with old method : 0.04903435707092285 length of segment : 302 time for calcul the mask position with numpy : 0.005466938018798828 nb_pixel_total : 134714 time to create 1 rle with old method : 0.15268683433532715 length of segment : 542 time for calcul the mask position with numpy : 0.005479335784912109 nb_pixel_total : 167843 time to create 1 rle with new method : 0.008289098739624023 length of segment : 434 time for calcul the mask position with numpy : 0.00029468536376953125 nb_pixel_total : 6316 time to create 1 rle with old method : 0.007550716400146484 length of segment : 85 time for calcul the mask position with numpy : 0.0005900859832763672 nb_pixel_total : 18369 time to create 1 rle with old method : 0.021266460418701172 length of segment : 154 time for calcul the mask position with numpy : 0.00010228157043457031 nb_pixel_total : 1428 time to create 1 rle with old method : 0.0017385482788085938 length of segment : 50 time for calcul the mask position with numpy : 0.0010302066802978516 nb_pixel_total : 17087 time to create 1 rle with old method : 0.0182797908782959 length of segment : 327 time for calcul the mask position with numpy : 0.00023555755615234375 nb_pixel_total : 2383 time to create 1 rle with old method : 0.002809286117553711 length of segment : 105 time for calcul the mask position with numpy : 0.0027081966400146484 nb_pixel_total : 58041 time to create 1 rle with old method : 0.06174921989440918 length of segment : 493 time for calcul the mask position with numpy : 0.006127357482910156 nb_pixel_total : 258806 time to create 1 rle with new method : 0.010241985321044922 length of segment : 638 time for calcul the mask position with numpy : 0.0004849433898925781 nb_pixel_total : 11348 time to create 1 rle with old method : 0.01263284683227539 length of segment : 260 time for calcul the mask position with numpy : 0.003177642822265625 nb_pixel_total : 121622 time to create 1 rle with old method : 0.12877106666564941 length of segment : 512 time for calcul the mask position with numpy : 0.0035958290100097656 nb_pixel_total : 228036 time to create 1 rle with new method : 0.009397506713867188 length of segment : 608 time for calcul the mask position with numpy : 0.0057866573333740234 nb_pixel_total : 161309 time to create 1 rle with new method : 0.007071733474731445 length of segment : 454 time for calcul the mask position with numpy : 0.0003361701965332031 nb_pixel_total : 4197 time to create 1 rle with old method : 0.005472898483276367 length of segment : 95 time for calcul the mask position with numpy : 0.00023674964904785156 nb_pixel_total : 4829 time to create 1 rle with old method : 0.005654573440551758 length of segment : 72 time for calcul the mask position with numpy : 0.006705284118652344 nb_pixel_total : 204498 time to create 1 rle with new method : 0.010035037994384766 length of segment : 764 time for calcul the mask position with numpy : 0.0017282962799072266 nb_pixel_total : 3680 time to create 1 rle with old method : 0.004532337188720703 length of segment : 115 time for calcul the mask position with numpy : 0.00047779083251953125 nb_pixel_total : 24750 time to create 1 rle with old method : 0.026564836502075195 length of segment : 258 time for calcul the mask position with numpy : 0.0003135204315185547 nb_pixel_total : 2113 time to create 1 rle with old method : 0.0025832653045654297 length of segment : 120 time for calcul the mask position with numpy : 0.0014202594757080078 nb_pixel_total : 44429 time to create 1 rle with old method : 0.04984879493713379 length of segment : 266 time for calcul the mask position with numpy : 0.0025255680084228516 nb_pixel_total : 110274 time to create 1 rle with old method : 0.11943888664245605 length of segment : 387 time for calcul the mask position with numpy : 0.0017843246459960938 nb_pixel_total : 66531 time to create 1 rle with old method : 0.07825589179992676 length of segment : 329 time for calcul the mask position with numpy : 0.0003483295440673828 nb_pixel_total : 5807 time to create 1 rle with old method : 0.007877349853515625 length of segment : 118 time for calcul the mask position with numpy : 0.0048749446868896484 nb_pixel_total : 128816 time to create 1 rle with old method : 0.1421666145324707 length of segment : 461 time for calcul the mask position with numpy : 0.0002110004425048828 nb_pixel_total : 3350 time to create 1 rle with old method : 0.00421142578125 length of segment : 55 time for calcul the mask position with numpy : 0.00318145751953125 nb_pixel_total : 112376 time to create 1 rle with old method : 0.1199338436126709 length of segment : 374 time for calcul the mask position with numpy : 0.00031447410583496094 nb_pixel_total : 5698 time to create 1 rle with old method : 0.006489276885986328 length of segment : 86 time for calcul the mask position with numpy : 0.005202054977416992 nb_pixel_total : 192090 time to create 1 rle with new method : 0.008229732513427734 length of segment : 552 time for calcul the mask position with numpy : 0.0015599727630615234 nb_pixel_total : 42982 time to create 1 rle with old method : 0.04732966423034668 length of segment : 274 time for calcul the mask position with numpy : 0.003893136978149414 nb_pixel_total : 174924 time to create 1 rle with new method : 0.00650477409362793 length of segment : 544 time for calcul the mask position with numpy : 0.0014231204986572266 nb_pixel_total : 43981 time to create 1 rle with old method : 0.04887104034423828 length of segment : 261 time for calcul the mask position with numpy : 0.0010662078857421875 nb_pixel_total : 29666 time to create 1 rle with old method : 0.03277111053466797 length of segment : 187 time for calcul the mask position with numpy : 0.001842498779296875 nb_pixel_total : 86859 time to create 1 rle with old method : 0.0940237045288086 length of segment : 796 time for calcul the mask position with numpy : 0.00022840499877929688 nb_pixel_total : 4464 time to create 1 rle with old method : 0.0053768157958984375 length of segment : 78 time for calcul the mask position with numpy : 0.0039103031158447266 nb_pixel_total : 160691 time to create 1 rle with new method : 0.006523847579956055 length of segment : 639 time for calcul the mask position with numpy : 0.008184671401977539 nb_pixel_total : 377761 time to create 1 rle with new method : 0.016995906829833984 length of segment : 567 time for calcul the mask position with numpy : 0.0003399848937988281 nb_pixel_total : 11466 time to create 1 rle with old method : 0.013041973114013672 length of segment : 118 time for calcul the mask position with numpy : 0.0017240047454833984 nb_pixel_total : 48085 time to create 1 rle with old method : 0.05344581604003906 length of segment : 317 time for calcul the mask position with numpy : 0.004709005355834961 nb_pixel_total : 313379 time to create 1 rle with new method : 0.009277582168579102 length of segment : 417 time for calcul the mask position with numpy : 9.059906005859375e-05 nb_pixel_total : 1661 time to create 1 rle with old method : 0.0022287368774414062 length of segment : 29 time for calcul the mask position with numpy : 0.0003457069396972656 nb_pixel_total : 7306 time to create 1 rle with old method : 0.008827686309814453 length of segment : 120 time for calcul the mask position with numpy : 0.0018777847290039062 nb_pixel_total : 58685 time to create 1 rle with old method : 0.0649714469909668 length of segment : 387 time for calcul the mask position with numpy : 0.00019741058349609375 nb_pixel_total : 4759 time to create 1 rle with old method : 0.0054852962493896484 length of segment : 59 time for calcul the mask position with numpy : 0.02571725845336914 nb_pixel_total : 1329008 time to create 1 rle with new method : 0.2543017864227295 length of segment : 2023 time for calcul the mask position with numpy : 0.0018932819366455078 nb_pixel_total : 73550 time to create 1 rle with old method : 0.0794515609741211 length of segment : 419 time for calcul the mask position with numpy : 0.00015997886657714844 nb_pixel_total : 2590 time to create 1 rle with old method : 0.0028412342071533203 length of segment : 90 time for calcul the mask position with numpy : 0.00021696090698242188 nb_pixel_total : 4745 time to create 1 rle with old method : 0.005935192108154297 length of segment : 64 time for calcul the mask position with numpy : 0.016111135482788086 nb_pixel_total : 750809 time to create 1 rle with new method : 0.06728839874267578 length of segment : 1074 time for calcul the mask position with numpy : 0.00019741058349609375 nb_pixel_total : 4004 time to create 1 rle with old method : 0.004570960998535156 length of segment : 99 time for calcul the mask position with numpy : 0.0004184246063232422 nb_pixel_total : 25794 time to create 1 rle with old method : 0.027857303619384766 length of segment : 214 time for calcul the mask position with numpy : 8.821487426757812e-05 nb_pixel_total : 2786 time to create 1 rle with old method : 0.003556489944458008 length of segment : 87 time for calcul the mask position with numpy : 0.00031948089599609375 nb_pixel_total : 7889 time to create 1 rle with old method : 0.009381771087646484 length of segment : 208 time for calcul the mask position with numpy : 0.00012874603271484375 nb_pixel_total : 3112 time to create 1 rle with old method : 0.003972053527832031 length of segment : 53 time spent for convertir_results : 22.331772565841675 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 140 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Number RLEs to save : 55308 save missing photos in datou_result : time spend for datou_step_exec : 106.6388955116272 time spend to save output : 20.381322383880615 total time spend for step 1 : 127.02021789550781 step2:crop_condition Fri Feb 7 06:40:36 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 140 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 34 About to insert : list_path_to_insert length 34 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1335348482, 1335348434, 1335348430, 1335348426, 1335348417, 1335348413, 1335193060, 1335193055, 1335193052, 1335193049] Looping around the photos to save general results len do output : 74 /-3659055276Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055265Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055324Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502575Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502576Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502616Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502621Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502637Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502634Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055272Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055293Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055313Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055320Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502588Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502603Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502630Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502631Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055274Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055291Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055287Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055310Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055306Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502589Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502605Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502611Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502625Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055270Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055277Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055275Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055267Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055278Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055268Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055262Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055281Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055288Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055297Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055280Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055292Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055283Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055304Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055309Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055303Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055305Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055316Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055315Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055323Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502590Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502582Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502579Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502608Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502613Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502593Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502598Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055327Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502609Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502627Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502629Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502628Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502624Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502642Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055271Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055286Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055290Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055302Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502585Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502604Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502599Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502614Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502623Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055318Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502622Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055296Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3659055311Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502596Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348482', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348434', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348430', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348426', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348417', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348413', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193060', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193055', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193052', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193049', None, None, None, None, None, '2561530') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 232 time used for this insertion : 0.05922222137451172 save_final save missing photos in datou_result : time spend for datou_step_exec : 43.030174016952515 time spend to save output : 0.07881760597229004 total time spend for step 2 : 43.108991622924805 step3:thcl Fri Feb 7 06:41:19 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.008443832397460938 time to convert the images to numpy array : 0.01569819450378418 time to import caffe and check if the image exist : 0.01239919662475586 time to convert the images to numpy array : 0.24135923385620117 time to import caffe and check if the image exist : 0.014088153839111328 time to convert the images to numpy array : 0.27653050422668457 time to import caffe and check if the image exist : 0.006524324417114258 time to convert the images to numpy array : 0.4372117519378662 time to import caffe and check if the image exist : 0.013520240783691406 time to convert the images to numpy array : 0.43023180961608887 time to import caffe and check if the image exist : 0.013452291488647461 time to convert the images to numpy array : 0.43483543395996094 time to import caffe and check if the image exist : 0.014609098434448242 time to convert the images to numpy array : 0.5551474094390869 time to import caffe and check if the image exist : 0.011096000671386719 time to convert the images to numpy array : 0.5713248252868652 time to import caffe and check if the image exist : 0.013872146606445312 time to convert the images to numpy array : 0.6018054485321045 time to import caffe and check if the image exist : 0.007050514221191406 time to convert the images to numpy array : 0.6143112182617188 total time to convert the images to numpy array : 0.9129414558410645 list photo_ids error: [] list photo_ids correct : [-3659055311, -3658502596, -3658502634, -3659055272, -3659055293, -3659055313, -3659055320, -3658502588, -3658502603, -3658502630, -3658502629, -3658502628, -3658502624, -3658502642, -3659055271, -3659055286, -3659055290, -3659055302, -3659055276, -3659055265, -3659055324, -3658502575, -3658502576, -3658502616, -3658502621, -3658502637, -3659055262, -3659055281, -3659055288, -3659055297, -3659055280, -3659055292, -3659055283, -3659055304, -3659055309, -3659055303, -3659055305, -3659055316, -3659055315, -3659055323, -3658502590, -3658502582, -3658502631, -3659055274, -3659055291, -3659055287, -3659055310, -3659055306, -3658502589, -3658502605, -3658502611, -3658502625, -3659055270, -3659055277, -3659055275, -3659055267, -3659055278, -3659055268, -3658502585, -3658502604, -3658502599, -3658502614, -3658502623, -3659055318, -3658502622, -3659055296, -3658502579, -3658502608, -3658502613, -3658502593, -3658502598, -3659055327, -3658502609, -3658502627] number of photos to traite : 74 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 962 wait 20 seconds l 3637 free memory gpu now : 962 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 524 wait 20 seconds l 3637 free memory gpu now : 524 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 2.06911301612854 time used to do the prediction : 0.2264707088470459 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 741 wait 20 seconds l 3637 free memory gpu now : 741 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 741 wait 20 seconds l 3637 free memory gpu now : 741 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.4555795192718506 time used to do the prediction : 0.18345308303833008 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 0.000751495361328125 save missing photos in datou_result : time spend for datou_step_exec : 92.84117245674133 time spend to save output : 0.005832195281982422 total time spend for step 3 : 92.84700465202332 step4:merge_mask_thcl_custom Fri Feb 7 06:42:52 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 140 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1335348482, 1335348434, 1335348430, 1335348426, 1335348417, 1335348413, 1335193060, 1335193055, 1335193052, 1335193049] Looping around the photos to save general results len do output : 10 /1335348482Didn't retrieve data .Didn't retrieve data . /1335348434Didn't retrieve data .Didn't retrieve data . /1335348430Didn't retrieve data .Didn't retrieve data . /1335348426Didn't retrieve data .Didn't retrieve data . /1335348417Didn't retrieve data .Didn't retrieve data . /1335348413Didn't retrieve data .Didn't retrieve data . /1335193060Didn't retrieve data .Didn't retrieve data . /1335193055Didn't retrieve data .Didn't retrieve data . /1335193052Didn't retrieve data .Didn't retrieve data . /1335193049Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348482', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348434', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348430', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348426', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348417', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348413', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193060', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193055', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193052', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193049', None, None, None, None, None, '2561530') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.10083770751953125 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.0694429874420166 time spend to save output : 0.10269474983215332 total time spend for step 4 : 0.17213773727416992 step5:rle_unique_nms_with_priority Fri Feb 7 06:42:52 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 18 nb_hashtags : 5 time to prepare the origin masks : 14.253937005996704 time for calcul the mask position with numpy : 0.6236741542816162 nb_pixel_total : 6140897 time to create 1 rle with new method : 0.7200069427490234 time for calcul the mask position with numpy : 0.056284427642822266 nb_pixel_total : 9965 time to create 1 rle with old method : 0.010769367218017578 time for calcul the mask position with numpy : 0.05588722229003906 nb_pixel_total : 14246 time to create 1 rle with old method : 0.017177820205688477 time for calcul the mask position with numpy : 0.05687761306762695 nb_pixel_total : 1638 time to create 1 rle with old method : 0.0019969940185546875 time for calcul the mask position with numpy : 0.05633902549743652 nb_pixel_total : 34517 time to create 1 rle with old method : 0.03854084014892578 time for calcul the mask position with numpy : 0.04946279525756836 nb_pixel_total : 58805 time to create 1 rle with old method : 0.06123661994934082 time for calcul the mask position with numpy : 0.050457000732421875 nb_pixel_total : 329893 time to create 1 rle with new method : 0.757901668548584 time for calcul the mask position with numpy : 0.05321168899536133 nb_pixel_total : 228657 time to create 1 rle with new method : 0.4790990352630615 time for calcul the mask position with numpy : 0.03113079071044922 nb_pixel_total : 58233 time to create 1 rle with old method : 0.06412720680236816 time for calcul the mask position with numpy : 0.029566049575805664 nb_pixel_total : 1385 time to create 1 rle with old method : 0.0017080307006835938 time for calcul the mask position with numpy : 0.02377605438232422 nb_pixel_total : 7671 time to create 1 rle with old method : 0.008816719055175781 time for calcul the mask position with numpy : 0.024401187896728516 nb_pixel_total : 43632 time to create 1 rle with old method : 0.047937870025634766 time for calcul the mask position with numpy : 0.02463221549987793 nb_pixel_total : 5381 time to create 1 rle with old method : 0.0062198638916015625 time for calcul the mask position with numpy : 0.023236751556396484 nb_pixel_total : 3183 time to create 1 rle with old method : 0.003622293472290039 time for calcul the mask position with numpy : 0.023342370986938477 nb_pixel_total : 33046 time to create 1 rle with old method : 0.035654544830322266 time for calcul the mask position with numpy : 0.022756576538085938 nb_pixel_total : 17685 time to create 1 rle with old method : 0.019711732864379883 time for calcul the mask position with numpy : 0.023508787155151367 nb_pixel_total : 5410 time to create 1 rle with old method : 0.00634455680847168 time for calcul the mask position with numpy : 0.02679896354675293 nb_pixel_total : 38947 time to create 1 rle with old method : 0.04285073280334473 time for calcul the mask position with numpy : 0.024606704711914062 nb_pixel_total : 17049 time to create 1 rle with old method : 0.02087116241455078 create new chi : 3.7474913597106934 time to delete rle : 0.012636899948120117 batch 1 Loaded 19 chid ids of type : 3726 Number RLEs to save : 11643 TO DO : save crop sub photo not yet done ! save time : 5.9871814250946045 nb_obj : 19 nb_hashtags : 5 time to prepare the origin masks : 12.505832433700562 time for calcul the mask position with numpy : 0.46843934059143066 nb_pixel_total : 6375393 time to create 1 rle with new method : 0.8401455879211426 time for calcul the mask position with numpy : 0.03577685356140137 nb_pixel_total : 763 time to create 1 rle with old method : 0.0014116764068603516 time for calcul the mask position with numpy : 0.037637948989868164 nb_pixel_total : 12949 time to create 1 rle with old method : 0.016791820526123047 time for calcul the mask position with numpy : 0.026441574096679688 nb_pixel_total : 96167 time to create 1 rle with old method : 0.12911152839660645 time for calcul the mask position with numpy : 0.028195619583129883 nb_pixel_total : 14337 time to create 1 rle with old method : 0.01869940757751465 time for calcul the mask position with numpy : 0.023686647415161133 nb_pixel_total : 1671 time to create 1 rle with old method : 0.0024127960205078125 time for calcul the mask position with numpy : 0.024622440338134766 nb_pixel_total : 1700 time to create 1 rle with old method : 0.002071857452392578 time for calcul the mask position with numpy : 0.0252530574798584 nb_pixel_total : 2586 time to create 1 rle with old method : 0.0031023025512695312 time for calcul the mask position with numpy : 0.024471759796142578 nb_pixel_total : 2506 time to create 1 rle with old method : 0.0030939579010009766 time for calcul the mask position with numpy : 0.02862095832824707 nb_pixel_total : 40555 time to create 1 rle with old method : 0.05053830146789551 time for calcul the mask position with numpy : 0.03464460372924805 nb_pixel_total : 10099 time to create 1 rle with old method : 0.012463808059692383 time for calcul the mask position with numpy : 0.030983686447143555 nb_pixel_total : 50255 time to create 1 rle with old method : 0.06049060821533203 time for calcul the mask position with numpy : 0.024163484573364258 nb_pixel_total : 12866 time to create 1 rle with old method : 0.01468348503112793 time for calcul the mask position with numpy : 0.02722454071044922 nb_pixel_total : 71544 time to create 1 rle with old method : 0.0813150405883789 time for calcul the mask position with numpy : 0.027938127517700195 nb_pixel_total : 109356 time to create 1 rle with old method : 0.12686538696289062 time for calcul the mask position with numpy : 0.024700641632080078 nb_pixel_total : 2614 time to create 1 rle with old method : 0.0031423568725585938 time for calcul the mask position with numpy : 0.03150510787963867 nb_pixel_total : 127239 time to create 1 rle with old method : 0.14561796188354492 time for calcul the mask position with numpy : 0.03140854835510254 nb_pixel_total : 108754 time to create 1 rle with old method : 0.12444114685058594 time for calcul the mask position with numpy : 0.028838396072387695 nb_pixel_total : 5341 time to create 1 rle with old method : 0.0066356658935546875 time for calcul the mask position with numpy : 0.030153512954711914 nb_pixel_total : 3545 time to create 1 rle with old method : 0.004325151443481445 create new chi : 2.7046213150024414 time to delete rle : 0.0011920928955078125 batch 1 Loaded 20 chid ids of type : 3726 Number RLEs to save : 9651 TO DO : save crop sub photo not yet done ! save time : 0.8030509948730469 nb_obj : 11 nb_hashtags : 5 time to prepare the origin masks : 6.665134906768799 time for calcul the mask position with numpy : 0.44449496269226074 nb_pixel_total : 4834943 time to create 1 rle with new method : 0.6559159755706787 time for calcul the mask position with numpy : 0.027239561080932617 nb_pixel_total : 1472 time to create 1 rle with old method : 0.0019469261169433594 time for calcul the mask position with numpy : 0.03105330467224121 nb_pixel_total : 25807 time to create 1 rle with old method : 0.029138803482055664 time for calcul the mask position with numpy : 0.03716421127319336 nb_pixel_total : 7036 time to create 1 rle with old method : 0.046555280685424805 time for calcul the mask position with numpy : 0.029946565628051758 nb_pixel_total : 247 time to create 1 rle with old method : 0.0004565715789794922 time for calcul the mask position with numpy : 0.02784442901611328 nb_pixel_total : 178342 time to create 1 rle with new method : 0.6735398769378662 time for calcul the mask position with numpy : 0.0381622314453125 nb_pixel_total : 95211 time to create 1 rle with old method : 0.11069273948669434 time for calcul the mask position with numpy : 0.044829607009887695 nb_pixel_total : 1534 time to create 1 rle with old method : 0.0018858909606933594 time for calcul the mask position with numpy : 0.04456639289855957 nb_pixel_total : 49805 time to create 1 rle with old method : 0.06443190574645996 time for calcul the mask position with numpy : 0.04302406311035156 nb_pixel_total : 39155 time to create 1 rle with old method : 0.044827938079833984 time for calcul the mask position with numpy : 0.07184624671936035 nb_pixel_total : 1522793 time to create 1 rle with new method : 0.5856733322143555 time for calcul the mask position with numpy : 0.03894948959350586 nb_pixel_total : 293895 time to create 1 rle with new method : 0.47254085540771484 create new chi : 3.6903603076934814 time to delete rle : 0.001402139663696289 batch 1 Loaded 12 chid ids of type : 3726 Number RLEs to save : 11673 TO DO : save crop sub photo not yet done ! save time : 1.092381238937378 nb_obj : 2 nb_hashtags : 2 time to prepare the origin masks : 3.4191834926605225 time for calcul the mask position with numpy : 0.32549428939819336 nb_pixel_total : 3736571 time to create 1 rle with new method : 0.4932236671447754 time for calcul the mask position with numpy : 0.026653528213500977 nb_pixel_total : 9287 time to create 1 rle with old method : 0.016445398330688477 time for calcul the mask position with numpy : 0.22768115997314453 nb_pixel_total : 3304382 time to create 1 rle with new method : 0.41135358810424805 create new chi : 1.5672030448913574 time to delete rle : 0.0008077621459960938 batch 1 Loaded 3 chid ids of type : 3726 Number RLEs to save : 9264 TO DO : save crop sub photo not yet done ! save time : 0.8491008281707764 nb_obj : 6 nb_hashtags : 4 time to prepare the origin masks : 5.473567724227905 time for calcul the mask position with numpy : 0.4721677303314209 nb_pixel_total : 3233132 time to create 1 rle with new method : 0.7702693939208984 time for calcul the mask position with numpy : 0.03861117362976074 nb_pixel_total : 204738 time to create 1 rle with new method : 0.545198917388916 time for calcul the mask position with numpy : 0.031150102615356445 nb_pixel_total : 31584 time to create 1 rle with old method : 0.03851175308227539 time for calcul the mask position with numpy : 0.0349123477935791 nb_pixel_total : 3070 time to create 1 rle with old method : 0.0037202835083007812 time for calcul the mask position with numpy : 0.14816761016845703 nb_pixel_total : 3515895 time to create 1 rle with new method : 0.8412885665893555 time for calcul the mask position with numpy : 0.022066354751586914 nb_pixel_total : 5939 time to create 1 rle with old method : 0.007372856140136719 time for calcul the mask position with numpy : 0.021718978881835938 nb_pixel_total : 55882 time to create 1 rle with old method : 0.0627443790435791 create new chi : 3.146279811859131 time to delete rle : 0.001451253890991211 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 9499 TO DO : save crop sub photo not yet done ! save time : 1.574141025543213 nb_obj : 3 nb_hashtags : 2 time to prepare the origin masks : 3.0477540493011475 time for calcul the mask position with numpy : 0.03882336616516113 nb_pixel_total : 2571512 time to create 1 rle with new method : 1.3812410831451416 time for calcul the mask position with numpy : 0.023275136947631836 nb_pixel_total : 65169 time to create 1 rle with old method : 0.08118367195129395 time for calcul the mask position with numpy : 0.023061037063598633 nb_pixel_total : 8367 time to create 1 rle with old method : 0.009740352630615234 time for calcul the mask position with numpy : 0.19257235527038574 nb_pixel_total : 4405192 time to create 1 rle with new method : 0.8866569995880127 create new chi : 2.7001986503601074 time to delete rle : 0.0007407665252685547 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 7950 TO DO : save crop sub photo not yet done ! save time : 0.8538455963134766 nb_obj : 18 nb_hashtags : 6 time to prepare the origin masks : 13.003106355667114 time for calcul the mask position with numpy : 0.23279142379760742 nb_pixel_total : 4238542 time to create 1 rle with new method : 0.6436665058135986 time for calcul the mask position with numpy : 0.03137946128845215 nb_pixel_total : 17985 time to create 1 rle with old method : 0.021248817443847656 time for calcul the mask position with numpy : 0.03601646423339844 nb_pixel_total : 183153 time to create 1 rle with new method : 0.5356512069702148 time for calcul the mask position with numpy : 0.03496599197387695 nb_pixel_total : 28297 time to create 1 rle with old method : 0.03282642364501953 time for calcul the mask position with numpy : 0.03464078903198242 nb_pixel_total : 228418 time to create 1 rle with new method : 0.46559572219848633 time for calcul the mask position with numpy : 0.03622889518737793 nb_pixel_total : 86598 time to create 1 rle with old method : 0.10291790962219238 time for calcul the mask position with numpy : 0.03639054298400879 nb_pixel_total : 8420 time to create 1 rle with old method : 0.010367155075073242 time for calcul the mask position with numpy : 0.03794717788696289 nb_pixel_total : 68373 time to create 1 rle with old method : 0.0864722728729248 time for calcul the mask position with numpy : 0.035759925842285156 nb_pixel_total : 1236 time to create 1 rle with old method : 0.001561880111694336 time for calcul the mask position with numpy : 0.035431623458862305 nb_pixel_total : 27407 time to create 1 rle with old method : 0.03495073318481445 time for calcul the mask position with numpy : 0.03798961639404297 nb_pixel_total : 150174 time to create 1 rle with new method : 1.4516568183898926 time for calcul the mask position with numpy : 0.046399593353271484 nb_pixel_total : 520612 time to create 1 rle with new method : 1.4663376808166504 time for calcul the mask position with numpy : 0.03919792175292969 nb_pixel_total : 22843 time to create 1 rle with old method : 0.04176044464111328 time for calcul the mask position with numpy : 0.07477259635925293 nb_pixel_total : 1400100 time to create 1 rle with new method : 1.7709672451019287 time for calcul the mask position with numpy : 0.04220700263977051 nb_pixel_total : 11479 time to create 1 rle with old method : 0.01611614227294922 time for calcul the mask position with numpy : 0.036266326904296875 nb_pixel_total : 5830 time to create 1 rle with old method : 0.006994009017944336 time for calcul the mask position with numpy : 0.0387117862701416 nb_pixel_total : 35228 time to create 1 rle with old method : 0.04142498970031738 time for calcul the mask position with numpy : 0.038147926330566406 nb_pixel_total : 7304 time to create 1 rle with old method : 0.008890390396118164 time for calcul the mask position with numpy : 0.03427863121032715 nb_pixel_total : 8241 time to create 1 rle with old method : 0.010178565979003906 create new chi : 7.867642402648926 time to delete rle : 0.003930568695068359 batch 1 Loaded 19 chid ids of type : 3726 Number RLEs to save : 20532 TO DO : save crop sub photo not yet done ! save time : 8.822680711746216 nb_obj : 23 nb_hashtags : 6 time to prepare the origin masks : 15.621337175369263 time for calcul the mask position with numpy : 0.49155497550964355 nb_pixel_total : 5768477 time to create 1 rle with new method : 1.22641921043396 time for calcul the mask position with numpy : 0.026694297790527344 nb_pixel_total : 947 time to create 1 rle with old method : 0.0021402835845947266 time for calcul the mask position with numpy : 0.024552106857299805 nb_pixel_total : 24499 time to create 1 rle with old method : 0.10922980308532715 time for calcul the mask position with numpy : 0.02405405044555664 nb_pixel_total : 109315 time to create 1 rle with old method : 0.1263580322265625 time for calcul the mask position with numpy : 0.025812387466430664 nb_pixel_total : 18337 time to create 1 rle with old method : 0.033705949783325195 time for calcul the mask position with numpy : 0.024566173553466797 nb_pixel_total : 3669 time to create 1 rle with old method : 0.004361152648925781 time for calcul the mask position with numpy : 0.023424386978149414 nb_pixel_total : 2094 time to create 1 rle with old method : 0.002582073211669922 time for calcul the mask position with numpy : 0.022701501846313477 nb_pixel_total : 16489 time to create 1 rle with old method : 0.018778562545776367 time for calcul the mask position with numpy : 0.024262428283691406 nb_pixel_total : 160854 time to create 1 rle with new method : 0.516613245010376 time for calcul the mask position with numpy : 0.02285027503967285 nb_pixel_total : 4785 time to create 1 rle with old method : 0.0058116912841796875 time for calcul the mask position with numpy : 0.02182483673095703 nb_pixel_total : 4154 time to create 1 rle with old method : 0.004892826080322266 time for calcul the mask position with numpy : 0.02117323875427246 nb_pixel_total : 55088 time to create 1 rle with old method : 0.06299805641174316 time for calcul the mask position with numpy : 0.021701812744140625 nb_pixel_total : 14215 time to create 1 rle with old method : 0.016903400421142578 time for calcul the mask position with numpy : 0.024732589721679688 nb_pixel_total : 121210 time to create 1 rle with old method : 0.13584136962890625 time for calcul the mask position with numpy : 0.022498130798339844 nb_pixel_total : 44428 time to create 1 rle with old method : 0.04991006851196289 time for calcul the mask position with numpy : 0.02220916748046875 nb_pixel_total : 203569 time to create 1 rle with new method : 0.6965687274932861 time for calcul the mask position with numpy : 0.022107839584350586 nb_pixel_total : 11350 time to create 1 rle with old method : 0.014188528060913086 time for calcul the mask position with numpy : 0.024303197860717773 nb_pixel_total : 258293 time to create 1 rle with new method : 0.4493238925933838 time for calcul the mask position with numpy : 0.02083730697631836 nb_pixel_total : 2370 time to create 1 rle with old method : 0.0028667449951171875 time for calcul the mask position with numpy : 0.033220767974853516 nb_pixel_total : 1418 time to create 1 rle with old method : 0.001720428466796875 time for calcul the mask position with numpy : 0.0344853401184082 nb_pixel_total : 6277 time to create 1 rle with old method : 0.0072400569915771484 time for calcul the mask position with numpy : 0.0361485481262207 nb_pixel_total : 167175 time to create 1 rle with new method : 0.451061487197876 time for calcul the mask position with numpy : 0.0348353385925293 nb_pixel_total : 45300 time to create 1 rle with old method : 0.05360555648803711 time for calcul the mask position with numpy : 0.03450202941894531 nb_pixel_total : 5927 time to create 1 rle with old method : 0.006940126419067383 create new chi : 5.230114459991455 time to delete rle : 0.012372493743896484 batch 1 Loaded 25 chid ids of type : 3726 Number RLEs to save : 14735 TO DO : save crop sub photo not yet done ! save time : 9.591670036315918 nb_obj : 20 nb_hashtags : 8 time to prepare the origin masks : 12.08253002166748 time for calcul the mask position with numpy : 0.3661675453186035 nb_pixel_total : 5619781 time to create 1 rle with new method : 0.5395159721374512 time for calcul the mask position with numpy : 0.021236658096313477 nb_pixel_total : 128607 time to create 1 rle with old method : 0.1407790184020996 time for calcul the mask position with numpy : 0.020519018173217773 nb_pixel_total : 77779 time to create 1 rle with old method : 0.08970308303833008 time for calcul the mask position with numpy : 0.0206601619720459 nb_pixel_total : 7252 time to create 1 rle with old method : 0.008296728134155273 time for calcul the mask position with numpy : 0.020281076431274414 nb_pixel_total : 1653 time to create 1 rle with old method : 0.0019893646240234375 time for calcul the mask position with numpy : 0.02038097381591797 nb_pixel_total : 1055 time to create 1 rle with old method : 0.0015184879302978516 time for calcul the mask position with numpy : 0.022741317749023438 nb_pixel_total : 312537 time to create 1 rle with new method : 0.45264768600463867 time for calcul the mask position with numpy : 0.021323680877685547 nb_pixel_total : 160307 time to create 1 rle with new method : 0.47382688522338867 time for calcul the mask position with numpy : 0.02189183235168457 nb_pixel_total : 47926 time to create 1 rle with old method : 0.0540926456451416 time for calcul the mask position with numpy : 0.02094292640686035 nb_pixel_total : 4415 time to create 1 rle with old method : 0.005144834518432617 time for calcul the mask position with numpy : 0.02108621597290039 nb_pixel_total : 11413 time to create 1 rle with old method : 0.012973308563232422 time for calcul the mask position with numpy : 0.021026611328125 nb_pixel_total : 43253 time to create 1 rle with old method : 0.04928898811340332 time for calcul the mask position with numpy : 0.021660804748535156 nb_pixel_total : 192864 time to create 1 rle with new method : 1.0779106616973877 time for calcul the mask position with numpy : 0.022165775299072266 nb_pixel_total : 43887 time to create 1 rle with old method : 0.04988670349121094 time for calcul the mask position with numpy : 0.02205824851989746 nb_pixel_total : 5666 time to create 1 rle with old method : 0.006592750549316406 time for calcul the mask position with numpy : 0.022602081298828125 nb_pixel_total : 112160 time to create 1 rle with old method : 0.12646698951721191 time for calcul the mask position with numpy : 0.022283315658569336 nb_pixel_total : 29561 time to create 1 rle with old method : 0.035784244537353516 time for calcul the mask position with numpy : 0.025030851364135742 nb_pixel_total : 3323 time to create 1 rle with old method : 0.004029512405395508 time for calcul the mask position with numpy : 0.022321224212646484 nb_pixel_total : 66267 time to create 1 rle with old method : 0.07391214370727539 time for calcul the mask position with numpy : 0.02191781997680664 nb_pixel_total : 5796 time to create 1 rle with old method : 0.00670623779296875 time for calcul the mask position with numpy : 0.02263355255126953 nb_pixel_total : 174738 time to create 1 rle with new method : 2.1134791374206543 create new chi : 6.2802019119262695 time to delete rle : 0.0030024051666259766 batch 1 Loaded 21 chid ids of type : 3726 Number RLEs to save : 13959 TO DO : save crop sub photo not yet done ! save time : 2.499202251434326 nb_obj : 13 nb_hashtags : 3 time to prepare the origin masks : 8.040914535522461 time for calcul the mask position with numpy : 0.4718012809753418 nb_pixel_total : 4765878 time to create 1 rle with new method : 1.852968692779541 time for calcul the mask position with numpy : 0.04161667823791504 nb_pixel_total : 3082 time to create 1 rle with old method : 0.005036115646362305 time for calcul the mask position with numpy : 0.024785280227661133 nb_pixel_total : 7798 time to create 1 rle with old method : 0.009386062622070312 time for calcul the mask position with numpy : 0.024736881256103516 nb_pixel_total : 73209 time to create 1 rle with old method : 0.08283734321594238 time for calcul the mask position with numpy : 0.024095535278320312 nb_pixel_total : 2783 time to create 1 rle with old method : 0.0033562183380126953 time for calcul the mask position with numpy : 0.02335977554321289 nb_pixel_total : 3974 time to create 1 rle with old method : 0.005995988845825195 time for calcul the mask position with numpy : 0.024741649627685547 nb_pixel_total : 1821 time to create 1 rle with old method : 0.005200386047363281 time for calcul the mask position with numpy : 0.053182363510131836 nb_pixel_total : 748418 time to create 1 rle with new method : 0.7876007556915283 time for calcul the mask position with numpy : 0.0238344669342041 nb_pixel_total : 4712 time to create 1 rle with old method : 0.14977526664733887 time for calcul the mask position with numpy : 0.024230480194091797 nb_pixel_total : 58443 time to create 1 rle with old method : 0.0616910457611084 time for calcul the mask position with numpy : 0.02467036247253418 nb_pixel_total : 2573 time to create 1 rle with old method : 0.0030410289764404297 time for calcul the mask position with numpy : 0.023175477981567383 nb_pixel_total : 22427 time to create 1 rle with old method : 0.023708581924438477 time for calcul the mask position with numpy : 0.041899919509887695 nb_pixel_total : 1350390 time to create 1 rle with new method : 2.0053036212921143 time for calcul the mask position with numpy : 0.02377033233642578 nb_pixel_total : 4732 time to create 1 rle with old method : 0.005455970764160156 create new chi : 5.963882684707642 time to delete rle : 0.00118255615234375 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 11832 TO DO : save crop sub photo not yet done ! save time : 2.3717336654663086 map_output_result : {1335348482: (0.0, 'Should be the crop_list due to order', 0.0), 1335348434: (0.0, 'Should be the crop_list due to order', 0.0), 1335348430: (0.0, 'Should be the crop_list due to order', 0.0), 1335348426: (0.0, 'Should be the crop_list due to order', 0.0), 1335348417: (0.0, 'Should be the crop_list due to order', 0.0), 1335348413: (0.0, 'Should be the crop_list due to order', 0.0), 1335193060: (0.0, 'Should be the crop_list due to order', 0.0), 1335193055: (0.0, 'Should be the crop_list due to order', 0.0), 1335193052: (0.0, 'Should be the crop_list due to order', 0.0), 1335193049: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1335348482, 1335348434, 1335348430, 1335348426, 1335348417, 1335348413, 1335193060, 1335193055, 1335193052, 1335193049] Looping around the photos to save general results len do output : 10 /1335348482.Didn't retrieve data . /1335348434.Didn't retrieve data . /1335348430.Didn't retrieve data . /1335348426.Didn't retrieve data . /1335348417.Didn't retrieve data . /1335348413.Didn't retrieve data . /1335193060.Didn't retrieve data . /1335193055.Didn't retrieve data . /1335193052.Didn't retrieve data . /1335193049.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348482', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348434', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348430', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348426', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348417', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348413', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193060', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193055', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193052', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193049', None, None, None, None, None, '2561530') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.053704023361206055 save_final save missing photos in datou_result : time spend for datou_step_exec : 173.18188738822937 time spend to save output : 0.062348127365112305 total time spend for step 5 : 173.24423551559448 step6:crop_condition Fri Feb 7 06:45:46 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 144 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738907150_2101163 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.5798308849334717 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! About to upload 5 photos upload in portfolio : 4869462 init cache_photo without model_param we have 5 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738907155_2101163 we have uploaded 5 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.4684369564056396 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738907160_2101163 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.7814221382141113 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738907166_2101163 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.6154096126556396 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! About to upload 7 photos upload in portfolio : 4869462 init cache_photo without model_param we have 7 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738907171_2101163 we have uploaded 7 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.661571502685547 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738907177_2101163 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.2242207527160645 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738907180_2101163 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6115484237670898 we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1335348482, 1335348434, 1335348430, 1335348426, 1335348417, 1335348413, 1335193060, 1335193055, 1335193052, 1335193049] Looping around the photos to save general results len do output : 33 /1335414341Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414343Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414345Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414347Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414372Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414373Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414374Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414375Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414376Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414383Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414384Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414385Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414386Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414387Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414388Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414390Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414391Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414392Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414393Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414394Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414395Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414396Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414397Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414398Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414399Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414400Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414401Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414402Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414404Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414405Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414406Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414407Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335414408Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348482', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348434', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348430', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348426', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348417', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348413', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193060', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193055', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193052', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193049', None, None, None, None, None, '2561530') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 109 time used for this insertion : 0.0360715389251709 save_final save missing photos in datou_result : time spend for datou_step_exec : 37.27196502685547 time spend to save output : 0.08269095420837402 total time spend for step 6 : 37.35465598106384 step7:ventilate_hashtags_in_portfolio Fri Feb 7 06:46:23 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20321739 get user id for portfolio 20321739 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20321739 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Teint_Dans_La_Masse','environnement','flou','Carton_gris','metal','autre_refus','Carton_brun','papier','plastique','kraft','mal_croppe','cartonnette')) AND mptpi.`min_score`=0.5 To do To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20321739 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Teint_Dans_La_Masse','environnement','flou','Carton_gris','metal','autre_refus','Carton_brun','papier','plastique','kraft','mal_croppe','cartonnette')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20321947,20321948,20321949,20321950,20321951,20321953,20321954,20321955,20321956,20321957,20321958,20321959?tags=environnement,autre_refus,Carton_brun,cartonnette,mal_croppe,metal,papier,flou,Carton_gris,Teint_Dans_La_Masse,kraft,plastique Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1335348482, 1335348434, 1335348430, 1335348426, 1335348417, 1335348413, 1335193060, 1335193055, 1335193052, 1335193049] Looping around the photos to save general results len do output : 1 /20321739. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348482', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348434', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348430', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348426', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348417', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348413', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193060', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193055', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193052', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193049', None, None, None, None, None, '2561530') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.01889777183532715 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.617077112197876 time spend to save output : 0.01917290687561035 total time spend for step 7 : 0.6362500190734863 step8:final Fri Feb 7 06:46:24 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1335348482: ('0.08843691174237035',), 1335348434: ('0.08843691174237035',), 1335348430: ('0.08843691174237035',), 1335348426: ('0.08843691174237035',), 1335348417: ('0.08843691174237035',), 1335348413: ('0.08843691174237035',), 1335193060: ('0.08843691174237035',), 1335193055: ('0.08843691174237035',), 1335193052: ('0.08843691174237035',), 1335193049: ('0.08843691174237035',)} new output for save of step final : {1335348482: ('0.08843691174237035',), 1335348434: ('0.08843691174237035',), 1335348430: ('0.08843691174237035',), 1335348426: ('0.08843691174237035',), 1335348417: ('0.08843691174237035',), 1335348413: ('0.08843691174237035',), 1335193060: ('0.08843691174237035',), 1335193055: ('0.08843691174237035',), 1335193052: ('0.08843691174237035',), 1335193049: ('0.08843691174237035',)} [1335348482, 1335348434, 1335348430, 1335348426, 1335348417, 1335348413, 1335193060, 1335193055, 1335193052, 1335193049] Looping around the photos to save general results len do output : 10 /1335348482.Didn't retrieve data . /1335348434.Didn't retrieve data . /1335348430.Didn't retrieve data . /1335348426.Didn't retrieve data . /1335348417.Didn't retrieve data . /1335348413.Didn't retrieve data . /1335193060.Didn't retrieve data . /1335193055.Didn't retrieve data . /1335193052.Didn't retrieve data . /1335193049.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348482', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348434', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348430', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348426', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348417', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348413', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193060', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193055', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193052', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193049', None, None, None, None, None, '2561530') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.04807591438293457 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.16137456893920898 time spend to save output : 0.04886364936828613 total time spend for step 8 : 0.21023821830749512 step9:velours_tree Fri Feb 7 06:46:24 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.2967214584350586 time spend to save output : 3.552436828613281e-05 total time spend for step 9 : 0.2967569828033447 step10:send_mail_cod Fri Feb 7 06:46:24 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20321739_07-02-2025_06_46_24.pdf 20321948 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203219481738907184 20321950 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203219501738907186 20321951 imagette203219511738907188 20321953 imagette203219531738907188 20321955 imagette203219551738907188 20321958 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203219581738907188 20321959 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203219591738907188 20321949 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203219491738907191 20321956 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203219561738907193 20321957 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203219571738907194 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20321739 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20321947,20321948,20321949,20321950,20321951,20321953,20321954,20321955,20321956,20321957,20321958,20321959?tags=environnement,autre_refus,Carton_brun,cartonnette,mal_croppe,metal,papier,flou,Carton_gris,Teint_Dans_La_Masse,kraft,plastique your option no_mail is active, we will not send the real mail to your client args[1335348482] : ('0.08843691174237035',) no score found for photo 1335348482 apple ('0.08843691174237035',) We are sending mail with results at report@fotonower.com args[1335348434] : ('0.08843691174237035',) no score found for photo 1335348434 apple ('0.08843691174237035',) We are sending mail with results at report@fotonower.com args[1335348430] : ('0.08843691174237035',) no score found for photo 1335348430 apple ('0.08843691174237035',) We are sending mail with results at report@fotonower.com args[1335348426] : ('0.08843691174237035',) no score found for photo 1335348426 apple ('0.08843691174237035',) We are sending mail with results at report@fotonower.com args[1335348417] : ('0.08843691174237035',) no score found for photo 1335348417 apple ('0.08843691174237035',) We are sending mail with results at report@fotonower.com args[1335348413] : ('0.08843691174237035',) no score found for photo 1335348413 apple ('0.08843691174237035',) We are sending mail with results at report@fotonower.com args[1335193060] : ('0.08843691174237035',) no score found for photo 1335193060 apple ('0.08843691174237035',) We are sending mail with results at report@fotonower.com args[1335193055] : ('0.08843691174237035',) no score found for photo 1335193055 apple ('0.08843691174237035',) We are sending mail with results at report@fotonower.com args[1335193052] : ('0.08843691174237035',) no score found for photo 1335193052 apple ('0.08843691174237035',) We are sending mail with results at report@fotonower.com args[1335193049] : ('0.08843691174237035',) no score found for photo 1335193049 apple ('0.08843691174237035',) We are sending mail with results at report@fotonower.com refus_total : 0.08843691174237035 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20321739 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1335191696,1335352200,1335350972,1335350967,1335348492,1335348488,1335193060,1335193055,1335193052,1335193049,1335193011,1335193003,1335192939,1335192936,1335192933,1335192926,1335191693,1335191945,1335192938,1335191988) Found this number of photos: 20 begin to download photo : 1335191696 begin to download photo : 1335348488 begin to download photo : 1335193011 begin to download photo : 1335192926 download finish for photo 1335348488 begin to download photo : 1335193060 download finish for photo 1335193011 begin to download photo : 1335193003 download finish for photo 1335191696 begin to download photo : 1335352200 download finish for photo 1335192926 begin to download photo : 1335191693 download finish for photo 1335193060 begin to download photo : 1335193055 download finish for photo 1335352200 begin to download photo : 1335350972 download finish for photo 1335191693 begin to download photo : 1335191945 download finish for photo 1335193003 begin to download photo : 1335192939 download finish for photo 1335193055 begin to download photo : 1335193052 download finish for photo 1335350972 begin to download photo : 1335350967 download finish for photo 1335191945 begin to download photo : 1335192938 download finish for photo 1335192939 begin to download photo : 1335192936 download finish for photo 1335193052 begin to download photo : 1335193049 download finish for photo 1335350967 begin to download photo : 1335348492 download finish for photo 1335192938 begin to download photo : 1335191988 download finish for photo 1335192936 begin to download photo : 1335192933 download finish for photo 1335193049 download finish for photo 1335191988 download finish for photo 1335348492 download finish for photo 1335192933 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20321739_07-02-2025_06_46_24.pdf results_Auto_P20321739_07-02-2025_06_46_24.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20321739_07-02-2025_06_46_24.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20321739','results_Auto_P20321739_07-02-2025_06_46_24.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20321739_07-02-2025_06_46_24.pdf','pdf','','1.59','0.08843691174237035') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1335348482, 1335348434, 1335348430, 1335348426, 1335348417, 1335348413, 1335193060, 1335193055, 1335193052, 1335193049] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348482', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348434', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348430', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348426', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348417', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348413', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193060', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193055', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193052', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193049', None, None, None, None, None, '2561530') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.014190196990966797 save_final save missing photos in datou_result : time spend for datou_step_exec : 16.517699241638184 time spend to save output : 0.014484405517578125 total time spend for step 10 : 16.53218364715576 step11:split_time_score Fri Feb 7 06:46:41 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('10', 37),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 06022025 20321739 Nombre de photos uploadées : 37 / 23040 (0%) 06022025 20321739 Nombre de photos taguées (types de déchets): 0 / 37 (0%) 06022025 20321739 Nombre de photos taguées (volume) : 0 / 37 (0%) elapsed_time : load_data_split_time_score 1.9073486328125e-06 elapsed_time : order_list_meta_photo_and_scores 6.198883056640625e-06 ????????????????????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0022869110107421875 elapsed_time : insert_dashboard_record_day_entry 0.030501842498779297 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.21863425925925936 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314812_07-02-2025_02_52_49.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314812 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314812 AND mptpi.`type`=3594 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314813 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314814 order by id desc limit 1 Qualite : 0.5683061661597331 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314815_07-02-2025_06_09_57.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314815 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11548 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11556 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11557 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11552 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11552 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11555 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11555 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11580 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11580 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11551 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11550 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11550 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11559 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11551 doesn't seem to be define in the database( WARNING : type of input 3 of step 11550 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11548 doesn't seem to be define in the database( WARNING : type of input 2 of step 11552 doesn't seem to be define in the database( WARNING : output 1 of step 11548 have datatype=2 whereas input 1 of step 11555 have datatype=7 WARNING : type of output 2 of step 11555 doesn't seem to be define in the database( WARNING : type of input 1 of step 11549 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11551 have datatype=10 whereas input 3 of step 11558 have datatype=6 WARNING : type of input 2 of step 11580 doesn't seem to be define in the database( WARNING : output 1 of step 11549 have datatype=7 whereas input 2 of step 11580 have datatype=None WARNING : type of output 3 of step 11580 doesn't seem to be define in the database( WARNING : type of input 1 of step 11551 doesn't seem to be define in the database( WARNING : output 0 of step 11551 have datatype=10 whereas input 0 of step 11586 have datatype=18 WARNING : type of input 5 of step 11558 doesn't seem to be define in the database( WARNING : output 0 of step 11586 have datatype=11 whereas input 5 of step 11558 have datatype=None WARNING : type of output 1 of step 11556 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : type of output 1 of step 11557 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : output 0 of step 11555 have datatype=1 whereas input 0 of step 11549 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314815 AND mptpi.`type`=4200 To do Qualite : 0.09494094623872477 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314816_07-02-2025_06_39_31.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314816 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314816 AND mptpi.`type`=4207 To do Qualite : 0.06847575159241935 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314817_07-02-2025_05_20_41.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314817 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314817 AND mptpi.`type`=4207 To do Qualite : 0.05500822396038016 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314818_07-02-2025_05_06_55.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314818 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314818 AND mptpi.`type`=4207 To do Qualite : 0.07107308905535348 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314819_07-02-2025_06_01_38.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314819 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314819 AND mptpi.`type`=4207 To do Qualite : 0.052388112823274276 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314820_07-02-2025_04_29_38.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314820 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314820 AND mptpi.`type`=4207 To do Qualite : 0.3475520682200876 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314821_07-02-2025_03_08_37.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314821 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11548 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11556 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11557 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11552 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11552 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11555 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11555 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11580 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11580 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11551 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11550 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11550 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11559 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11551 doesn't seem to be define in the database( WARNING : type of input 3 of step 11550 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11548 doesn't seem to be define in the database( WARNING : type of input 2 of step 11552 doesn't seem to be define in the database( WARNING : output 1 of step 11548 have datatype=2 whereas input 1 of step 11555 have datatype=7 WARNING : type of output 2 of step 11555 doesn't seem to be define in the database( WARNING : type of input 1 of step 11549 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11551 have datatype=10 whereas input 3 of step 11558 have datatype=6 WARNING : type of input 2 of step 11580 doesn't seem to be define in the database( WARNING : output 1 of step 11549 have datatype=7 whereas input 2 of step 11580 have datatype=None WARNING : type of output 3 of step 11580 doesn't seem to be define in the database( WARNING : type of input 1 of step 11551 doesn't seem to be define in the database( WARNING : output 0 of step 11551 have datatype=10 whereas input 0 of step 11586 have datatype=18 WARNING : type of input 5 of step 11558 doesn't seem to be define in the database( WARNING : output 0 of step 11586 have datatype=11 whereas input 5 of step 11558 have datatype=None WARNING : type of output 1 of step 11556 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : type of output 1 of step 11557 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : output 0 of step 11555 have datatype=1 whereas input 0 of step 11549 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314821 AND mptpi.`type`=4200 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20321707 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20321708 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20321709 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20321710 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20321711 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20321736 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20321737 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20321738 order by id desc limit 1 Qualite : 0.08940862040233692 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20321739_07-02-2025_06_46_24.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20321739 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20321739 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20321740 order by id desc limit 1 Qualite : 0.07606678211572258 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314829_07-02-2025_03_59_22.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314829 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314829 AND mptpi.`type`=4207 To do Qualite : 0.10086758606120957 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314830_07-02-2025_04_18_00.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314830 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314830 AND mptpi.`type`=4207 To do Qualite : 0.06528518323435197 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314831_07-02-2025_03_09_05.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314831 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314831 AND mptpi.`type`=4207 To do Qualite : 0.08062900675827052 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314832_07-02-2025_03_50_17.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314832 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314832 AND mptpi.`type`=4207 To do Qualite : 0.053841303219029274 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314833_07-02-2025_02_40_39.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314833 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314833 AND mptpi.`type`=4207 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314834 order by id desc limit 1 Qualite : 0.22870597929691197 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314835_07-02-2025_02_26_31.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314835 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314835 AND mptpi.`type`=3594 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20321741 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20321742 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20321743 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20321744 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20321745 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20321746 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20321748 order by id desc limit 1 Qualite : 0.05675741930944238 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20321749_07-02-2025_06_46_40.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20321749 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20321749 AND mptpi.`type`=4207 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20321750 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20321752 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20321753 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314843 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20321754 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314845 order by id desc limit 1 Qualite : 0.08977715635017561 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314846_07-02-2025_01_55_45.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314846 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314846 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314847 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314848 order by id desc limit 1 Qualite : 0.0951918982153927 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314849_07-02-2025_03_54_57.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314849 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314849 AND mptpi.`type`=4207 To do Qualite : 0.048795321359665696 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314850_07-02-2025_03_49_26.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314850 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314850 AND mptpi.`type`=4207 To do Qualite : 0.0794257709299137 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314851_07-02-2025_02_06_41.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314851 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314851 AND mptpi.`type`=4207 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314852 order by id desc limit 1 Qualite : 0.08573655308919938 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314853_07-02-2025_01_44_23.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314853 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314853 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314854 order by id desc limit 1 Qualite : 0.2245652913943355 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314855_07-02-2025_01_50_41.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314855 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314855 AND mptpi.`type`=3594 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314856 order by id desc limit 1 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314857_07-02-2025_02_05_12.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314857 order by id desc limit 1 NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'06022025': {'nb_upload': 37, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1335348482, 1335348434, 1335348430, 1335348426, 1335348417, 1335348413, 1335193060, 1335193055, 1335193052, 1335193049] Looping around the photos to save general results len do output : 1 /20321739Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348482', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348434', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348430', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348426', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348417', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335348413', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193060', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193055', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193052', None, None, None, None, None, '2561530') ('3459', None, None, None, None, None, None, None, '2561530') ('3459', None, '1335193049', None, None, None, None, None, '2561530') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.02419447898864746 save_final save missing photos in datou_result : time spend for datou_step_exec : 26.64850640296936 time spend to save output : 0.024441242218017578 total time spend for step 11 : 26.672947645187378 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 151.51user 174.13system 8:45.59elapsed 61%CPU (0avgtext+0avgdata 4307480maxresident)k 1179704inputs+184984outputs (23279major+12882501minor)pagefaults 0swaps