python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 1785227 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 30>10 please execute split_portfolio.py -i 20970847 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2628787'] with mtr_portfolio_ids : ['20970847'] and first list_photo_ids : [] new path : /proc/1785227/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 1.110447645187378 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Mon Mar 3 11:14:27 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 7037 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-03-03 11:14:30.701194: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-03 11:14:30.731065: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-03 11:14:30.733179: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f1bb0000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-03 11:14:30.733230: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-03 11:14:30.736924: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-03 11:14:30.984164: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1e7b75f0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-03 11:14:30.984215: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-03 11:14:30.985464: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-03 11:14:30.985904: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-03 11:14:30.988852: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-03 11:14:30.991431: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-03 11:14:30.991934: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-03 11:14:30.994104: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-03 11:14:30.995183: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-03 11:14:30.999342: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-03 11:14:31.000587: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-03 11:14:31.000670: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-03 11:14:31.001325: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-03 11:14:31.001340: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-03 11:14:31.001349: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-03 11:14:31.002500: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6472 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-03-03 11:14:31.264059: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-03 11:14:31.264148: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-03 11:14:31.264168: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-03 11:14:31.264187: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-03 11:14:31.264205: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-03 11:14:31.264223: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-03 11:14:31.264241: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-03 11:14:31.264259: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-03 11:14:31.265664: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-03 11:14:31.266864: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-03 11:14:31.266917: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-03 11:14:31.266939: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-03 11:14:31.266957: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-03 11:14:31.266974: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-03 11:14:31.266992: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-03 11:14:31.267014: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-03 11:14:31.267035: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-03 11:14:31.268433: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-03 11:14:31.268466: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-03 11:14:31.268477: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-03 11:14:31.268486: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-03 11:14:31.269874: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6472 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-03-03 11:14:38.860457: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-03 11:14:39.055495: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 6 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 6 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 9 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (1080, 1920, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 1920.00000 nb d'objets trouves : 5 Detection mask done ! Trying to reset tf kernel 1785614 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 1748 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 7037 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.0010578632354736328 nb_pixel_total : 62029 time to create 1 rle with old method : 0.06605291366577148 length of segment : 336 time for calcul the mask position with numpy : 0.012294530868530273 nb_pixel_total : 567454 time to create 1 rle with new method : 0.06392550468444824 length of segment : 1081 time for calcul the mask position with numpy : 0.0004000663757324219 nb_pixel_total : 20956 time to create 1 rle with old method : 0.024851322174072266 length of segment : 220 time for calcul the mask position with numpy : 0.0001418590545654297 nb_pixel_total : 5816 time to create 1 rle with old method : 0.007312774658203125 length of segment : 63 time for calcul the mask position with numpy : 0.00019025802612304688 nb_pixel_total : 3569 time to create 1 rle with old method : 0.004317283630371094 length of segment : 198 time for calcul the mask position with numpy : 0.0007851123809814453 nb_pixel_total : 46561 time to create 1 rle with old method : 0.053208112716674805 length of segment : 529 time for calcul the mask position with numpy : 0.00026869773864746094 nb_pixel_total : 11676 time to create 1 rle with old method : 0.013870477676391602 length of segment : 182 time for calcul the mask position with numpy : 0.00010728836059570312 nb_pixel_total : 3308 time to create 1 rle with old method : 0.004027843475341797 length of segment : 60 time for calcul the mask position with numpy : 0.010331153869628906 nb_pixel_total : 595721 time to create 1 rle with new method : 0.04987192153930664 length of segment : 1252 time for calcul the mask position with numpy : 0.0006442070007324219 nb_pixel_total : 34911 time to create 1 rle with old method : 0.04704999923706055 length of segment : 252 time for calcul the mask position with numpy : 0.00018715858459472656 nb_pixel_total : 6978 time to create 1 rle with old method : 0.010284662246704102 length of segment : 69 time for calcul the mask position with numpy : 0.012902259826660156 nb_pixel_total : 615063 time to create 1 rle with new method : 0.045735836029052734 length of segment : 1605 time for calcul the mask position with numpy : 0.003947019577026367 nb_pixel_total : 300266 time to create 1 rle with new method : 0.010951519012451172 length of segment : 927 time for calcul the mask position with numpy : 0.0011239051818847656 nb_pixel_total : 61350 time to create 1 rle with old method : 0.09479761123657227 length of segment : 265 time for calcul the mask position with numpy : 0.0016236305236816406 nb_pixel_total : 87017 time to create 1 rle with old method : 0.09554290771484375 length of segment : 638 time for calcul the mask position with numpy : 0.015332221984863281 nb_pixel_total : 627786 time to create 1 rle with new method : 0.14632964134216309 length of segment : 1496 time for calcul the mask position with numpy : 0.0051691532135009766 nb_pixel_total : 356891 time to create 1 rle with new method : 0.012985706329345703 length of segment : 948 time for calcul the mask position with numpy : 0.00146484375 nb_pixel_total : 92368 time to create 1 rle with old method : 0.10174274444580078 length of segment : 367 time for calcul the mask position with numpy : 9.059906005859375e-05 nb_pixel_total : 1078 time to create 1 rle with old method : 0.0012547969818115234 length of segment : 99 time for calcul the mask position with numpy : 0.0004372596740722656 nb_pixel_total : 27553 time to create 1 rle with old method : 0.030123233795166016 length of segment : 418 time for calcul the mask position with numpy : 0.0013346672058105469 nb_pixel_total : 71578 time to create 1 rle with old method : 0.08173441886901855 length of segment : 826 time for calcul the mask position with numpy : 0.013941526412963867 nb_pixel_total : 811345 time to create 1 rle with new method : 0.06076335906982422 length of segment : 1418 time for calcul the mask position with numpy : 0.00011396408081054688 nb_pixel_total : 3481 time to create 1 rle with old method : 0.0044651031494140625 length of segment : 73 time for calcul the mask position with numpy : 0.0006775856018066406 nb_pixel_total : 48071 time to create 1 rle with old method : 0.055960655212402344 length of segment : 195 time for calcul the mask position with numpy : 0.003893613815307617 nb_pixel_total : 308737 time to create 1 rle with new method : 0.010186195373535156 length of segment : 980 time for calcul the mask position with numpy : 0.00032067298889160156 nb_pixel_total : 17808 time to create 1 rle with old method : 0.02279496192932129 length of segment : 171 time for calcul the mask position with numpy : 0.0002727508544921875 nb_pixel_total : 8041 time to create 1 rle with old method : 0.009709835052490234 length of segment : 85 time for calcul the mask position with numpy : 0.01331329345703125 nb_pixel_total : 703349 time to create 1 rle with new method : 0.06755924224853516 length of segment : 1855 time for calcul the mask position with numpy : 0.0012748241424560547 nb_pixel_total : 21817 time to create 1 rle with old method : 0.02779531478881836 length of segment : 168 time for calcul the mask position with numpy : 0.001958131790161133 nb_pixel_total : 61452 time to create 1 rle with old method : 0.07109665870666504 length of segment : 463 time for calcul the mask position with numpy : 0.0005304813385009766 nb_pixel_total : 15195 time to create 1 rle with old method : 0.017689943313598633 length of segment : 194 time for calcul the mask position with numpy : 0.000377655029296875 nb_pixel_total : 8650 time to create 1 rle with old method : 0.010251522064208984 length of segment : 254 time for calcul the mask position with numpy : 0.00038242340087890625 nb_pixel_total : 12253 time to create 1 rle with old method : 0.014509439468383789 length of segment : 117 time for calcul the mask position with numpy : 0.0005283355712890625 nb_pixel_total : 12659 time to create 1 rle with old method : 0.015461444854736328 length of segment : 161 time for calcul the mask position with numpy : 0.024163246154785156 nb_pixel_total : 977677 time to create 1 rle with new method : 0.19196772575378418 length of segment : 954 time for calcul the mask position with numpy : 0.0006182193756103516 nb_pixel_total : 42092 time to create 1 rle with old method : 0.049590349197387695 length of segment : 309 time for calcul the mask position with numpy : 0.00016880035400390625 nb_pixel_total : 3024 time to create 1 rle with old method : 0.0037620067596435547 length of segment : 57 time for calcul the mask position with numpy : 0.00013637542724609375 nb_pixel_total : 7540 time to create 1 rle with old method : 0.009522199630737305 length of segment : 82 time for calcul the mask position with numpy : 0.004378318786621094 nb_pixel_total : 296014 time to create 1 rle with new method : 0.008173942565917969 length of segment : 882 time for calcul the mask position with numpy : 0.0006043910980224609 nb_pixel_total : 13610 time to create 1 rle with old method : 0.016014814376831055 length of segment : 269 time for calcul the mask position with numpy : 0.013897418975830078 nb_pixel_total : 722699 time to create 1 rle with new method : 0.06062626838684082 length of segment : 1215 time for calcul the mask position with numpy : 0.00022268295288085938 nb_pixel_total : 1315 time to create 1 rle with old method : 0.002382993698120117 length of segment : 64 time for calcul the mask position with numpy : 0.004846811294555664 nb_pixel_total : 380592 time to create 1 rle with new method : 0.011807680130004883 length of segment : 1113 time for calcul the mask position with numpy : 0.0003712177276611328 nb_pixel_total : 18283 time to create 1 rle with old method : 0.021515369415283203 length of segment : 231 time for calcul the mask position with numpy : 0.0010232925415039062 nb_pixel_total : 75061 time to create 1 rle with old method : 0.10339522361755371 length of segment : 437 time for calcul the mask position with numpy : 0.0010159015655517578 nb_pixel_total : 76000 time to create 1 rle with old method : 0.08842277526855469 length of segment : 443 time for calcul the mask position with numpy : 0.00029730796813964844 nb_pixel_total : 17722 time to create 1 rle with old method : 0.021150827407836914 length of segment : 153 time for calcul the mask position with numpy : 0.0012352466583251953 nb_pixel_total : 38076 time to create 1 rle with old method : 0.04291820526123047 length of segment : 436 time for calcul the mask position with numpy : 0.007365703582763672 nb_pixel_total : 378233 time to create 1 rle with new method : 0.1241767406463623 length of segment : 1656 time for calcul the mask position with numpy : 0.001672983169555664 nb_pixel_total : 43379 time to create 1 rle with old method : 0.04826951026916504 length of segment : 596 time for calcul the mask position with numpy : 0.00034880638122558594 nb_pixel_total : 14126 time to create 1 rle with old method : 0.023562908172607422 length of segment : 122 time for calcul the mask position with numpy : 0.0007238388061523438 nb_pixel_total : 39374 time to create 1 rle with old method : 0.04481077194213867 length of segment : 492 time for calcul the mask position with numpy : 0.0005421638488769531 nb_pixel_total : 26299 time to create 1 rle with old method : 0.03024768829345703 length of segment : 398 time for calcul the mask position with numpy : 7.081031799316406e-05 nb_pixel_total : 1130 time to create 1 rle with old method : 0.0015418529510498047 length of segment : 27 time for calcul the mask position with numpy : 0.04919886589050293 nb_pixel_total : 1170028 time to create 1 rle with new method : 0.06255388259887695 length of segment : 1734 time for calcul the mask position with numpy : 0.0025453567504882812 nb_pixel_total : 193718 time to create 1 rle with new method : 0.006112575531005859 length of segment : 635 time for calcul the mask position with numpy : 0.0020949840545654297 nb_pixel_total : 111054 time to create 1 rle with old method : 0.13028192520141602 length of segment : 626 time for calcul the mask position with numpy : 0.0047550201416015625 nb_pixel_total : 307617 time to create 1 rle with new method : 0.01129293441772461 length of segment : 952 time spent for convertir_results : 7.1295294761657715 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 72 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Number RLEs to save : 31848 save missing photos in datou_result : time spend for datou_step_exec : 27.674901008605957 time spend to save output : 1.9243528842926025 total time spend for step 1 : 29.59925389289856 step2:crop_condition Mon Mar 3 11:14:57 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 72 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1340605978, 1340605880, 1340605874, 1340605798, 1340605788, 1340605740, 1340605737, 1340605689, 1340605686, 1340605623] Looping around the photos to save general results len do output : 16 /-3692254238Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692254245Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692254220Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692280709Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692280708Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692254215Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692280716Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692254227Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692254235Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692254233Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692254240Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692254197Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692280712Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692254203Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692280718Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3692254216Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605978', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605880', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605874', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605798', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605788', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605740', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605737', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605689', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605686', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605623', None, None, None, None, None, '2628787') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 58 time used for this insertion : 0.020966529846191406 save_final save missing photos in datou_result : time spend for datou_step_exec : 3.9508419036865234 time spend to save output : 0.02207469940185547 total time spend for step 2 : 3.972916603088379 step3:thcl Mon Mar 3 11:15:01 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.003384113311767578 time to convert the images to numpy array : 5.0067901611328125e-06 time to import caffe and check if the image exist : 0.004497528076171875 time to convert the images to numpy array : 2.384185791015625e-06 time to import caffe and check if the image exist : 0.006316184997558594 time to convert the images to numpy array : 0.010152578353881836 time to import caffe and check if the image exist : 0.007200956344604492 time to convert the images to numpy array : 0.01030278205871582 time to import caffe and check if the image exist : 0.007218837738037109 time to convert the images to numpy array : 0.011202335357666016 time to import caffe and check if the image exist : 0.005329608917236328 time to convert the images to numpy array : 0.013405561447143555 time to import caffe and check if the image exist : 0.004128694534301758 time to convert the images to numpy array : 0.017090320587158203 time to import caffe and check if the image exist : 0.003789663314819336 time to convert the images to numpy array : 0.018437862396240234 time to import caffe and check if the image exist : 0.007012605667114258 time to convert the images to numpy array : 0.019065380096435547 time to import caffe and check if the image exist : 0.0035963058471679688 time to convert the images to numpy array : 0.026991605758666992 total time to convert the images to numpy array : 0.26494574546813965 list photo_ids error: [] list photo_ids correct : [-3692254240, -3692254197, -3692254220, -3692280709, -3692280718, -3692254216, -3692280708, -3692254215, -3692280716, -3692254227, -3692254235, -3692254233, -3692280712, -3692254203, -3692254238, -3692254245] number of photos to traite : 16 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10814 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10595 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.17453622817993164 time used to do the prediction : 0.2881932258605957 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10593 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10593 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.17441320419311523 time used to do the prediction : 0.05613827705383301 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 1.1205673217773438e-05 save missing photos in datou_result : time spend for datou_step_exec : 7.855807065963745 time spend to save output : 0.0003025531768798828 total time spend for step 3 : 7.856109619140625 step4:merge_mask_thcl_custom Mon Mar 3 11:15:09 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 72 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1340605978, 1340605880, 1340605874, 1340605798, 1340605788, 1340605740, 1340605737, 1340605689, 1340605686, 1340605623] Looping around the photos to save general results len do output : 10 /1340605978Didn't retrieve data .Didn't retrieve data . /1340605880Didn't retrieve data .Didn't retrieve data . /1340605874Didn't retrieve data .Didn't retrieve data . /1340605798Didn't retrieve data .Didn't retrieve data . /1340605788Didn't retrieve data .Didn't retrieve data . /1340605740Didn't retrieve data .Didn't retrieve data . /1340605737Didn't retrieve data .Didn't retrieve data . /1340605689Didn't retrieve data .Didn't retrieve data . /1340605686Didn't retrieve data .Didn't retrieve data . /1340605623Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605978', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605880', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605874', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605798', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605788', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605740', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605737', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605689', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605686', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605623', None, None, None, None, None, '2628787') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.014009714126586914 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.054775238037109375 time spend to save output : 0.014616250991821289 total time spend for step 4 : 0.06939148902893066 step5:rle_unique_nms_with_priority Mon Mar 3 11:15:09 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 4 nb_hashtags : 2 time to prepare the origin masks : 0.7742612361907959 time for calcul the mask position with numpy : 0.04572749137878418 nb_pixel_total : 1416134 time to create 1 rle with new method : 0.18967080116271973 time for calcul the mask position with numpy : 0.010108709335327148 nb_pixel_total : 22411 time to create 1 rle with old method : 0.024343252182006836 time for calcul the mask position with numpy : 0.006203413009643555 nb_pixel_total : 5760 time to create 1 rle with old method : 0.0063419342041015625 time for calcul the mask position with numpy : 0.009737014770507812 nb_pixel_total : 567296 time to create 1 rle with new method : 0.09626412391662598 time for calcul the mask position with numpy : 0.0070264339447021484 nb_pixel_total : 61999 time to create 1 rle with old method : 0.0679330825805664 create new chi : 0.48057103157043457 time to delete rle : 0.02191305160522461 batch 1 Loaded 5 chid ids of type : 3726 Number RLEs to save : 4200 TO DO : save crop sub photo not yet done ! save time : 0.2752385139465332 nb_obj : 6 nb_hashtags : 3 time to prepare the origin masks : 0.899259090423584 time for calcul the mask position with numpy : 0.06856441497802734 nb_pixel_total : 1347314 time to create 1 rle with new method : 0.10897159576416016 time for calcul the mask position with numpy : 0.011259794235229492 nb_pixel_total : 34863 time to create 1 rle with old method : 0.04006052017211914 time for calcul the mask position with numpy : 0.0064885616302490234 nb_pixel_total : 1382 time to create 1 rle with old method : 0.0022749900817871094 time for calcul the mask position with numpy : 0.014606714248657227 nb_pixel_total : 628679 time to create 1 rle with new method : 0.0997161865234375 time for calcul the mask position with numpy : 0.006020069122314453 nb_pixel_total : 3291 time to create 1 rle with old method : 0.0038568973541259766 time for calcul the mask position with numpy : 0.006220102310180664 nb_pixel_total : 11634 time to create 1 rle with old method : 0.01304936408996582 time for calcul the mask position with numpy : 0.0063669681549072266 nb_pixel_total : 46437 time to create 1 rle with old method : 0.0501248836517334 create new chi : 0.45399022102355957 time to delete rle : 0.0005359649658203125 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 5815 TO DO : save crop sub photo not yet done ! save time : 0.3398270606994629 nb_obj : 5 nb_hashtags : 1 time to prepare the origin masks : 0.3473551273345947 time for calcul the mask position with numpy : 0.016375064849853516 nb_pixel_total : 1386354 time to create 1 rle with new method : 0.20166015625 time for calcul the mask position with numpy : 0.00651097297668457 nb_pixel_total : 234 time to create 1 rle with old method : 0.0007684230804443359 time for calcul the mask position with numpy : 0.007790803909301758 nb_pixel_total : 66458 time to create 1 rle with old method : 0.07895922660827637 time for calcul the mask position with numpy : 0.005738258361816406 nb_pixel_total : 2096 time to create 1 rle with old method : 0.003304004669189453 time for calcul the mask position with numpy : 0.010087728500366211 nb_pixel_total : 611494 time to create 1 rle with new method : 0.028856992721557617 time for calcul the mask position with numpy : 0.005896806716918945 nb_pixel_total : 6964 time to create 1 rle with old method : 0.00782012939453125 create new chi : 0.3798351287841797 time to delete rle : 0.0005970001220703125 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 6636 TO DO : save crop sub photo not yet done ! save time : 0.4125685691833496 nb_obj : 7 nb_hashtags : 2 time to prepare the origin masks : 0.9621171951293945 time for calcul the mask position with numpy : 0.04212379455566406 nb_pixel_total : 1137922 time to create 1 rle with new method : 0.19740724563598633 time for calcul the mask position with numpy : 0.006372928619384766 nb_pixel_total : 58163 time to create 1 rle with old method : 0.063140869140625 time for calcul the mask position with numpy : 0.0059070587158203125 nb_pixel_total : 1075 time to create 1 rle with old method : 0.0011882781982421875 time for calcul the mask position with numpy : 0.0060346126556396484 nb_pixel_total : 112 time to create 1 rle with old method : 0.0003714561462402344 time for calcul the mask position with numpy : 0.0057811737060546875 nb_pixel_total : 18720 time to create 1 rle with old method : 0.02083754539489746 time for calcul the mask position with numpy : 0.006378889083862305 nb_pixel_total : 89183 time to create 1 rle with old method : 0.09978795051574707 time for calcul the mask position with numpy : 0.011008739471435547 nb_pixel_total : 681313 time to create 1 rle with new method : 0.16322088241577148 time for calcul the mask position with numpy : 0.00642848014831543 nb_pixel_total : 87112 time to create 1 rle with old method : 0.09517788887023926 create new chi : 0.7462546825408936 time to delete rle : 0.0006461143493652344 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 6561 TO DO : save crop sub photo not yet done ! save time : 0.4063713550567627 nb_obj : 8 nb_hashtags : 3 time to prepare the origin masks : 0.8299911022186279 time for calcul the mask position with numpy : 0.0370023250579834 nb_pixel_total : 1074820 time to create 1 rle with new method : 0.20051789283752441 time for calcul the mask position with numpy : 0.006057024002075195 nb_pixel_total : 16851 time to create 1 rle with old method : 0.01917409896850586 time for calcul the mask position with numpy : 0.00650787353515625 nb_pixel_total : 34673 time to create 1 rle with old method : 0.0385432243347168 time for calcul the mask position with numpy : 0.006005048751831055 nb_pixel_total : 3464 time to create 1 rle with old method : 0.004044532775878906 time for calcul the mask position with numpy : 0.005939960479736328 nb_pixel_total : 3482 time to create 1 rle with old method : 0.004975318908691406 time for calcul the mask position with numpy : 0.011237859725952148 nb_pixel_total : 793171 time to create 1 rle with new method : 0.07887005805969238 time for calcul the mask position with numpy : 0.006313323974609375 nb_pixel_total : 71654 time to create 1 rle with old method : 0.08241081237792969 time for calcul the mask position with numpy : 0.006207466125488281 nb_pixel_total : 47917 time to create 1 rle with old method : 0.05188465118408203 time for calcul the mask position with numpy : 0.006052970886230469 nb_pixel_total : 27568 time to create 1 rle with old method : 0.030518054962158203 create new chi : 0.618614673614502 time to delete rle : 0.0007874965667724609 batch 1 Loaded 12 chid ids of type : 3726 Number RLEs to save : 7967 TO DO : save crop sub photo not yet done ! save time : 0.4767580032348633 nb_obj : 7 nb_hashtags : 3 time to prepare the origin masks : 0.7271747589111328 time for calcul the mask position with numpy : 0.0410771369934082 nb_pixel_total : 1263409 time to create 1 rle with new method : 0.1701653003692627 time for calcul the mask position with numpy : 0.006310462951660156 nb_pixel_total : 4726 time to create 1 rle with old method : 0.005744457244873047 time for calcul the mask position with numpy : 0.0062410831451416016 nb_pixel_total : 14933 time to create 1 rle with old method : 0.01682281494140625 time for calcul the mask position with numpy : 0.006096839904785156 nb_pixel_total : 761 time to create 1 rle with old method : 0.0011043548583984375 time for calcul the mask position with numpy : 0.0063631534576416016 nb_pixel_total : 36441 time to create 1 rle with old method : 0.041025638580322266 time for calcul the mask position with numpy : 0.005953311920166016 nb_pixel_total : 21416 time to create 1 rle with old method : 0.028998613357543945 time for calcul the mask position with numpy : 0.01010274887084961 nb_pixel_total : 21751 time to create 1 rle with old method : 0.023404836654663086 time for calcul the mask position with numpy : 0.019196271896362305 nb_pixel_total : 710163 time to create 1 rle with new method : 0.0790107250213623 create new chi : 0.4809443950653076 time to delete rle : 0.0006549358367919922 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 7457 TO DO : save crop sub photo not yet done ! save time : 0.43044185638427734 nb_obj : 9 nb_hashtags : 2 time to prepare the origin masks : 0.847440242767334 time for calcul the mask position with numpy : 0.03273916244506836 nb_pixel_total : 961909 time to create 1 rle with new method : 0.1924145221710205 time for calcul the mask position with numpy : 0.006311655044555664 nb_pixel_total : 42736 time to create 1 rle with old method : 0.04879617691040039 time for calcul the mask position with numpy : 0.006065845489501953 nb_pixel_total : 3008 time to create 1 rle with old method : 0.0034837722778320312 time for calcul the mask position with numpy : 0.005934238433837891 nb_pixel_total : 7504 time to create 1 rle with old method : 0.00836181640625 time for calcul the mask position with numpy : 0.0067615509033203125 nb_pixel_total : 42031 time to create 1 rle with old method : 0.0455625057220459 time for calcul the mask position with numpy : 0.0063838958740234375 nb_pixel_total : 5436 time to create 1 rle with old method : 0.007370471954345703 time for calcul the mask position with numpy : 0.01343226432800293 nb_pixel_total : 977548 time to create 1 rle with new method : 0.0918729305267334 time for calcul the mask position with numpy : 0.006739139556884766 nb_pixel_total : 12600 time to create 1 rle with old method : 0.019499778747558594 time for calcul the mask position with numpy : 0.0062830448150634766 nb_pixel_total : 12209 time to create 1 rle with old method : 0.013662099838256836 time for calcul the mask position with numpy : 0.006172895431518555 nb_pixel_total : 8619 time to create 1 rle with old method : 0.009281396865844727 create new chi : 0.5557849407196045 time to delete rle : 0.0007522106170654297 batch 1 Loaded 11 chid ids of type : 3726 Number RLEs to save : 6285 TO DO : save crop sub photo not yet done ! save time : 0.6189725399017334 nb_obj : 8 nb_hashtags : 2 time to prepare the origin masks : 0.9753391742706299 time for calcul the mask position with numpy : 0.04716324806213379 nb_pixel_total : 1158121 time to create 1 rle with new method : 0.27091240882873535 time for calcul the mask position with numpy : 0.0067234039306640625 nb_pixel_total : 17602 time to create 1 rle with old method : 0.02050471305847168 time for calcul the mask position with numpy : 0.010562658309936523 nb_pixel_total : 3670 time to create 1 rle with old method : 0.004282474517822266 time for calcul the mask position with numpy : 0.006644487380981445 nb_pixel_total : 74838 time to create 1 rle with old method : 0.08244919776916504 time for calcul the mask position with numpy : 0.006559848785400391 nb_pixel_total : 52264 time to create 1 rle with old method : 0.06212496757507324 time for calcul the mask position with numpy : 0.010073423385620117 nb_pixel_total : 1315 time to create 1 rle with old method : 0.0015759468078613281 time for calcul the mask position with numpy : 0.010762929916381836 nb_pixel_total : 23330 time to create 1 rle with old method : 0.02708601951599121 time for calcul the mask position with numpy : 0.020593881607055664 nb_pixel_total : 728887 time to create 1 rle with new method : 0.20232605934143066 time for calcul the mask position with numpy : 0.010257482528686523 nb_pixel_total : 13573 time to create 1 rle with old method : 0.015119552612304688 create new chi : 0.8291950225830078 time to delete rle : 0.0007417201995849609 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 7905 TO DO : save crop sub photo not yet done ! save time : 0.45401787757873535 nb_obj : 6 nb_hashtags : 2 time to prepare the origin masks : 1.0666861534118652 time for calcul the mask position with numpy : 0.024466276168823242 nb_pixel_total : 1576252 time to create 1 rle with new method : 0.37339091300964355 time for calcul the mask position with numpy : 0.006857633590698242 nb_pixel_total : 26454 time to create 1 rle with old method : 0.029596567153930664 time for calcul the mask position with numpy : 0.006734132766723633 nb_pixel_total : 157 time to create 1 rle with old method : 0.0003643035888671875 time for calcul the mask position with numpy : 0.00672459602355957 nb_pixel_total : 14011 time to create 1 rle with old method : 0.015799760818481445 time for calcul the mask position with numpy : 0.008747100830078125 nb_pixel_total : 375621 time to create 1 rle with new method : 0.14929556846618652 time for calcul the mask position with numpy : 0.00652003288269043 nb_pixel_total : 37903 time to create 1 rle with old method : 0.0429079532623291 time for calcul the mask position with numpy : 0.006392955780029297 nb_pixel_total : 43202 time to create 1 rle with old method : 0.04828763008117676 create new chi : 0.742713212966919 time to delete rle : 0.0007216930389404297 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 7072 TO DO : save crop sub photo not yet done ! save time : 0.9760160446166992 nb_obj : 5 nb_hashtags : 2 time to prepare the origin masks : 0.6514530181884766 time for calcul the mask position with numpy : 0.01507711410522461 nb_pixel_total : 644613 time to create 1 rle with new method : 0.3214223384857178 time for calcul the mask position with numpy : 0.011006355285644531 nb_pixel_total : 77297 time to create 1 rle with old method : 0.0887763500213623 time for calcul the mask position with numpy : 0.006947755813598633 nb_pixel_total : 106805 time to create 1 rle with old method : 0.12195730209350586 time for calcul the mask position with numpy : 0.006999492645263672 nb_pixel_total : 71231 time to create 1 rle with old method : 0.10212588310241699 time for calcul the mask position with numpy : 0.01021575927734375 nb_pixel_total : 1917 time to create 1 rle with old method : 0.003998756408691406 time for calcul the mask position with numpy : 0.06115365028381348 nb_pixel_total : 1171737 time to create 1 rle with new method : 0.5014772415161133 create new chi : 1.268188238143921 time to delete rle : 0.0018727779388427734 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 8013 TO DO : save crop sub photo not yet done ! save time : 0.5724751949310303 map_output_result : {1340605978: (0.0, 'Should be the crop_list due to order', 0), 1340605880: (0.0, 'Should be the crop_list due to order', 0), 1340605874: (0.0, 'Should be the crop_list due to order', 0), 1340605798: (0.0, 'Should be the crop_list due to order', 0), 1340605788: (0.0, 'Should be the crop_list due to order', 0), 1340605740: (0.0, 'Should be the crop_list due to order', 0), 1340605737: (0.0, 'Should be the crop_list due to order', 0), 1340605689: (0.0, 'Should be the crop_list due to order', 0), 1340605686: (0.0, 'Should be the crop_list due to order', 0), 1340605623: (0.0, 'Should be the crop_list due to order', 0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1340605978, 1340605880, 1340605874, 1340605798, 1340605788, 1340605740, 1340605737, 1340605689, 1340605686, 1340605623] Looping around the photos to save general results len do output : 10 /1340605978.Didn't retrieve data . /1340605880.Didn't retrieve data . /1340605874.Didn't retrieve data . /1340605798.Didn't retrieve data . /1340605788.Didn't retrieve data . /1340605740.Didn't retrieve data . /1340605737.Didn't retrieve data . /1340605689.Didn't retrieve data . /1340605686.Didn't retrieve data . /1340605623.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605978', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605880', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605874', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605798', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605788', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605740', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605737', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605689', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605686', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605623', None, None, None, None, None, '2628787') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.01312398910522461 save_final save missing photos in datou_result : time spend for datou_step_exec : 19.962199449539185 time spend to save output : 0.013720512390136719 total time spend for step 5 : 19.97591996192932 step6:crop_condition Mon Mar 3 11:15:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 84 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740996930_1785227 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.6591212749481201 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740996932_1785227 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.068519115447998 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1740996935_1785227 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.4469027519226074 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1340605978, 1340605880, 1340605874, 1340605798, 1340605788, 1340605740, 1340605737, 1340605689, 1340605686, 1340605623] Looping around the photos to save general results len do output : 6 /1340610148Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340610150Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340610151Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340610154Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340610155Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1340610156Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605978', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605880', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605874', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605798', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605788', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605740', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605737', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605689', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605686', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605623', None, None, None, None, None, '2628787') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 28 time used for this insertion : 0.023858308792114258 save_final save missing photos in datou_result : time spend for datou_step_exec : 7.427919864654541 time spend to save output : 0.024262189865112305 total time spend for step 6 : 7.452182054519653 step7:ventilate_hashtags_in_portfolio Mon Mar 3 11:15:36 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20970847 get user id for portfolio 20970847 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20970847 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Teint_Dans_La_Masse','environnement','metal','Carton_brun','kraft','cartonnette','plastique','Carton_gris','papier','flou','mal_croppe','autre_refus')) AND mptpi.`min_score`=0.5 To do To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20970847 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('Teint_Dans_La_Masse','environnement','metal','Carton_brun','kraft','cartonnette','plastique','Carton_gris','papier','flou','mal_croppe','autre_refus')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20971036,20971037,20971038,20971039,20971040,20971041,20971042,20971043,20971044,20971045,20971046,20971047?tags=mal_croppe,kraft,environnement,metal,cartonnette,Teint_Dans_La_Masse,papier,plastique,flou,autre_refus,Carton_brun,Carton_gris Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1340605978, 1340605880, 1340605874, 1340605798, 1340605788, 1340605740, 1340605737, 1340605689, 1340605686, 1340605623] Looping around the photos to save general results len do output : 1 /20970847. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605978', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605880', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605874', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605798', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605788', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605740', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605737', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605689', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605686', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605623', None, None, None, None, None, '2628787') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.01360321044921875 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.6994109153747559 time spend to save output : 0.013837099075317383 total time spend for step 7 : 0.7132480144500732 step8:final Mon Mar 3 11:15:37 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1340605978: ('0.02701664992347413',), 1340605880: ('0.02701664992347413',), 1340605874: ('0.02701664992347413',), 1340605798: ('0.02701664992347413',), 1340605788: ('0.02701664992347413',), 1340605740: ('0.02701664992347413',), 1340605737: ('0.02701664992347413',), 1340605689: ('0.02701664992347413',), 1340605686: ('0.02701664992347413',), 1340605623: ('0.02701664992347413',)} new output for save of step final : {1340605978: ('0.02701664992347413',), 1340605880: ('0.02701664992347413',), 1340605874: ('0.02701664992347413',), 1340605798: ('0.02701664992347413',), 1340605788: ('0.02701664992347413',), 1340605740: ('0.02701664992347413',), 1340605737: ('0.02701664992347413',), 1340605689: ('0.02701664992347413',), 1340605686: ('0.02701664992347413',), 1340605623: ('0.02701664992347413',)} [1340605978, 1340605880, 1340605874, 1340605798, 1340605788, 1340605740, 1340605737, 1340605689, 1340605686, 1340605623] Looping around the photos to save general results len do output : 10 /1340605978.Didn't retrieve data . /1340605880.Didn't retrieve data . /1340605874.Didn't retrieve data . /1340605798.Didn't retrieve data . /1340605788.Didn't retrieve data . /1340605740.Didn't retrieve data . /1340605737.Didn't retrieve data . /1340605689.Didn't retrieve data . /1340605686.Didn't retrieve data . /1340605623.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605978', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605880', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605874', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605798', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605788', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605740', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605737', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605689', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605686', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605623', None, None, None, None, None, '2628787') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013629436492919922 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.10198616981506348 time spend to save output : 0.014110088348388672 total time spend for step 8 : 0.11609625816345215 step9:velours_tree Mon Mar 3 11:15:37 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.032279014587402344 time spend to save output : 5.841255187988281e-05 total time spend for step 9 : 0.03233742713928223 step10:send_mail_cod Mon Mar 3 11:15:37 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20970847_03-03-2025_11_15_37.pdf 20971036 imagette209710361740996937 20971037 imagette209710371740996937 20971039 imagette209710391740996937 20971040 change filename to text .change filename to text .change filename to text .change filename to text .imagette209710401740996937 20971043 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette209710431740996937 20971044 imagette209710441740996938 20971045 change filename to text .change filename to text .change filename to text .imagette209710451740996938 20971041 change filename to text .change filename to text .change filename to text .imagette209710411740996938 20971046 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette209710461740996938 20971047 change filename to text .change filename to text .imagette209710471740996939 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20970847 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20971036,20971037,20971038,20971039,20971040,20971041,20971042,20971043,20971044,20971045,20971046,20971047?tags=mal_croppe,kraft,environnement,metal,cartonnette,Teint_Dans_La_Masse,papier,plastique,flou,autre_refus,Carton_brun,Carton_gris your option no_mail is active, we will not send the real mail to your client args[1340605978] : ('0.02701664992347413',) no score found for photo 1340605978 We are sending mail with results at report@fotonower.com args[1340605880] : ('0.02701664992347413',) no score found for photo 1340605880 We are sending mail with results at report@fotonower.com args[1340605874] : ('0.02701664992347413',) no score found for photo 1340605874 We are sending mail with results at report@fotonower.com args[1340605798] : ('0.02701664992347413',) no score found for photo 1340605798 We are sending mail with results at report@fotonower.com args[1340605788] : ('0.02701664992347413',) no score found for photo 1340605788 We are sending mail with results at report@fotonower.com args[1340605740] : ('0.02701664992347413',) no score found for photo 1340605740 We are sending mail with results at report@fotonower.com args[1340605737] : ('0.02701664992347413',) no score found for photo 1340605737 We are sending mail with results at report@fotonower.com args[1340605689] : ('0.02701664992347413',) no score found for photo 1340605689 We are sending mail with results at report@fotonower.com args[1340605686] : ('0.02701664992347413',) no score found for photo 1340605686 We are sending mail with results at report@fotonower.com args[1340605623] : ('0.02701664992347413',) no score found for photo 1340605623 We are sending mail with results at report@fotonower.com refus_total : 0.02701664992347413 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20970847 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1340606027,1340606225,1340606151,1340606639,1340606533,1340606520,1340606473,1340606470,1340606403,1340606400,1340606351,1340606347,1340606287,1340606283,1340606229,1340606181,1340605623,1340606086,1340606079,1340606032) Found this number of photos: 20 begin to download photo : 1340606027 begin to download photo : 1340606520 begin to download photo : 1340606351 begin to download photo : 1340606181 download finish for photo 1340606181 begin to download photo : 1340605623 download finish for photo 1340606027 begin to download photo : 1340606225 download finish for photo 1340606351 begin to download photo : 1340606347 download finish for photo 1340606520 begin to download photo : 1340606473 download finish for photo 1340606225 begin to download photo : 1340606151 download finish for photo 1340605623 begin to download photo : 1340606086 download finish for photo 1340606347 begin to download photo : 1340606287 download finish for photo 1340606473 begin to download photo : 1340606470 download finish for photo 1340606287 begin to download photo : 1340606283 download finish for photo 1340606086 begin to download photo : 1340606079 download finish for photo 1340606151 begin to download photo : 1340606639 download finish for photo 1340606470 begin to download photo : 1340606403 download finish for photo 1340606079 begin to download photo : 1340606032 download finish for photo 1340606283 begin to download photo : 1340606229 download finish for photo 1340606403 begin to download photo : 1340606400 download finish for photo 1340606639 begin to download photo : 1340606533 download finish for photo 1340606400 download finish for photo 1340606533 download finish for photo 1340606229 download finish for photo 1340606032 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20970847_03-03-2025_11_15_37.pdf results_Auto_P20970847_03-03-2025_11_15_37.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20970847_03-03-2025_11_15_37.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20970847','results_Auto_P20970847_03-03-2025_11_15_37.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20970847_03-03-2025_11_15_37.pdf','pdf','','0.27','0.02701664992347413') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1340605978, 1340605880, 1340605874, 1340605798, 1340605788, 1340605740, 1340605737, 1340605689, 1340605686, 1340605623] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605978', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605880', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605874', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605798', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605788', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605740', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605737', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605689', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605686', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605623', None, None, None, None, None, '2628787') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.01365041732788086 save_final save missing photos in datou_result : time spend for datou_step_exec : 3.675548553466797 time spend to save output : 0.013939857482910156 total time spend for step 10 : 3.689488410949707 step11:split_time_score Mon Mar 3 11:15:41 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('10', 30),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 03032025 20970847 Nombre de photos uploadées : 30 / 23040 (0%) 03032025 20970847 Nombre de photos taguées (types de déchets): 0 / 30 (0%) 03032025 20970847 Nombre de photos taguées (volume) : 0 / 30 (0%) elapsed_time : load_data_split_time_score 2.6226043701171875e-06 elapsed_time : order_list_meta_photo_and_scores 7.152557373046875e-06 ?????????????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0017008781433105469 elapsed_time : insert_dashboard_record_day_entry 0.022054195404052734 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.07226605924691387 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20964506_03-03-2025_08_42_01.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20964506 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20964506 AND mptpi.`type`=3726 To do Qualite : 0.07603242440997401 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20965613_03-03-2025_08_25_42.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20965613 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20965613 AND mptpi.`type`=3726 To do Qualite : 0.02575062473468879 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20965638_03-03-2025_08_15_41.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20965638 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20965638 AND mptpi.`type`=3726 To do Qualite : 0.05752190734891299 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20965639_03-03-2025_08_11_47.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20965639 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20965639 AND mptpi.`type`=3726 To do Qualite : 0.03688006233820538 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20966600_03-03-2025_09_11_26.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20966600 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20966600 AND mptpi.`type`=3726 To do Qualite : 0.02831058440182939 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20967444_03-03-2025_09_46_54.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20967444 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20967444 AND mptpi.`type`=3726 To do Qualite : 0.021500941685675855 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20967445_03-03-2025_09_40_48.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20967445 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20967445 AND mptpi.`type`=3726 To do Qualite : 0.01965095775535203 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20969599_03-03-2025_10_43_39.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20969599 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20969599 AND mptpi.`type`=3726 To do Qualite : 0.04866769047720368 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20969612_03-03-2025_10_52_27.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20969612 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20969612 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20969625 order by id desc limit 1 Qualite : 0.03740780134606926 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20969626_03-03-2025_10_43_47.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20969626 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20969626 AND mptpi.`type`=3726 To do Qualite : 0.024803821231226257 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20970847_03-03-2025_11_15_37.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20970847 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20970847 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20970848 order by id desc limit 1 NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'03032025': {'nb_upload': 30, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1340605978, 1340605880, 1340605874, 1340605798, 1340605788, 1340605740, 1340605737, 1340605689, 1340605686, 1340605623] Looping around the photos to save general results len do output : 1 /20970847Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605978', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605880', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605874', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605798', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605788', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605740', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605737', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605689', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605686', None, None, None, None, None, '2628787') ('3459', None, None, None, None, None, None, None, '2628787') ('3459', None, '1340605623', None, None, None, None, None, '2628787') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.013493061065673828 save_final save missing photos in datou_result : time spend for datou_step_exec : 2.3787975311279297 time spend to save output : 0.013829469680786133 total time spend for step 11 : 2.392627000808716 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 39.85user 26.27system 1:19.26elapsed 83%CPU (0avgtext+0avgdata 2754048maxresident)k 58168inputs+31216outputs (268major+2224506minor)pagefaults 0swaps