python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 376160 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 27>10 please execute split_portfolio.py -i 20423281 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2574201'] with mtr_portfolio_ids : ['20423281'] and first list_photo_ids : [] new path : /proc/376160/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 2.0626001358032227 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Tue Feb 11 00:04:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3584 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-11 00:04:34.461552: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-11 00:04:34.495254: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-11 00:04:34.498074: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f2110000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-11 00:04:34.498133: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-11 00:04:34.503519: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-11 00:04:34.760481: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xa41c7b0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-11 00:04:34.760558: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-11 00:04:34.762727: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-11 00:04:34.763235: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-11 00:04:34.767683: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-11 00:04:34.788244: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-11 00:04:34.788971: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-11 00:04:34.825920: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-11 00:04:34.828066: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-11 00:04:34.891624: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-11 00:04:34.892979: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-11 00:04:34.893480: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-11 00:04:34.894216: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-11 00:04:34.894236: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-11 00:04:34.894249: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-11 00:04:34.896212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3132 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-11 00:04:35.209689: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-11 00:04:35.209802: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-11 00:04:35.209824: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-11 00:04:35.209843: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-11 00:04:35.209862: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-11 00:04:35.209880: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-11 00:04:35.209898: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-11 00:04:35.209917: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-11 00:04:35.210986: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-11 00:04:35.212161: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-11 00:04:35.212196: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-11 00:04:35.212215: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-11 00:04:35.212233: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-11 00:04:35.212251: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-11 00:04:35.212269: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-11 00:04:35.212286: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-11 00:04:35.212303: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-11 00:04:35.213347: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-11 00:04:35.213384: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-11 00:04:35.213395: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-11 00:04:35.213405: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-11 00:04:35.214516: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3132 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-11 00:04:45.514011: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-11 00:04:45.745017: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-11 00:04:47.638471: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.56G (2749235200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-11 00:04:48.418512: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.67GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-11 00:04:48.418699: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.67GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-11 00:04:48.425422: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-11 00:04:48.425447: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 23 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 14 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 15 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 31 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 13 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 19 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 29 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 26 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 15 Detection mask done ! Trying to reset tf kernel 376533 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 31 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 3584 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.0008113384246826172 nb_pixel_total : 14468 time to create 1 rle with old method : 0.02164292335510254 length of segment : 311 time for calcul the mask position with numpy : 0.0007183551788330078 nb_pixel_total : 42061 time to create 1 rle with old method : 0.04294943809509277 length of segment : 250 time for calcul the mask position with numpy : 0.3343367576599121 nb_pixel_total : 1400465 time to create 1 rle with new method : 0.12224650382995605 length of segment : 2284 time for calcul the mask position with numpy : 0.0009224414825439453 nb_pixel_total : 40602 time to create 1 rle with old method : 0.04430747032165527 length of segment : 199 time for calcul the mask position with numpy : 0.0002129077911376953 nb_pixel_total : 8726 time to create 1 rle with old method : 0.009447813034057617 length of segment : 119 time for calcul the mask position with numpy : 6.341934204101562e-05 nb_pixel_total : 1196 time to create 1 rle with old method : 0.0014638900756835938 length of segment : 49 time for calcul the mask position with numpy : 0.00035190582275390625 nb_pixel_total : 17554 time to create 1 rle with old method : 0.018435001373291016 length of segment : 241 time for calcul the mask position with numpy : 0.0016720294952392578 nb_pixel_total : 108818 time to create 1 rle with old method : 0.11251163482666016 length of segment : 551 time for calcul the mask position with numpy : 0.0007100105285644531 nb_pixel_total : 40975 time to create 1 rle with old method : 0.04271745681762695 length of segment : 278 time for calcul the mask position with numpy : 0.0001361370086669922 nb_pixel_total : 4864 time to create 1 rle with old method : 0.0052564144134521484 length of segment : 78 time for calcul the mask position with numpy : 0.0067653656005859375 nb_pixel_total : 443411 time to create 1 rle with new method : 0.02753901481628418 length of segment : 945 time for calcul the mask position with numpy : 0.0018804073333740234 nb_pixel_total : 95696 time to create 1 rle with old method : 0.09977149963378906 length of segment : 410 time for calcul the mask position with numpy : 0.0007054805755615234 nb_pixel_total : 36919 time to create 1 rle with old method : 0.03956198692321777 length of segment : 264 time for calcul the mask position with numpy : 0.0008873939514160156 nb_pixel_total : 53881 time to create 1 rle with old method : 0.05730867385864258 length of segment : 317 time for calcul the mask position with numpy : 0.0011365413665771484 nb_pixel_total : 78676 time to create 1 rle with old method : 0.08502960205078125 length of segment : 376 time for calcul the mask position with numpy : 0.0017216205596923828 nb_pixel_total : 109256 time to create 1 rle with old method : 0.1168832778930664 length of segment : 472 time for calcul the mask position with numpy : 0.0015897750854492188 nb_pixel_total : 83260 time to create 1 rle with old method : 0.08730125427246094 length of segment : 573 time for calcul the mask position with numpy : 0.0013463497161865234 nb_pixel_total : 66200 time to create 1 rle with old method : 0.06896257400512695 length of segment : 386 time for calcul the mask position with numpy : 0.0013277530670166016 nb_pixel_total : 95881 time to create 1 rle with old method : 0.09818267822265625 length of segment : 358 time for calcul the mask position with numpy : 0.0020797252655029297 nb_pixel_total : 147053 time to create 1 rle with old method : 0.15056347846984863 length of segment : 479 time for calcul the mask position with numpy : 0.00012755393981933594 nb_pixel_total : 4627 time to create 1 rle with old method : 0.0055446624755859375 length of segment : 108 time for calcul the mask position with numpy : 0.0032927989959716797 nb_pixel_total : 248284 time to create 1 rle with new method : 0.008413314819335938 length of segment : 731 time for calcul the mask position with numpy : 0.00044608116149902344 nb_pixel_total : 34399 time to create 1 rle with old method : 0.037415504455566406 length of segment : 262 time for calcul the mask position with numpy : 0.00038933753967285156 nb_pixel_total : 17657 time to create 1 rle with old method : 0.01971912384033203 length of segment : 155 time for calcul the mask position with numpy : 0.0026061534881591797 nb_pixel_total : 156512 time to create 1 rle with new method : 0.007112979888916016 length of segment : 591 time for calcul the mask position with numpy : 7.271766662597656e-05 nb_pixel_total : 1747 time to create 1 rle with old method : 0.0020678043365478516 length of segment : 46 time for calcul the mask position with numpy : 7.176399230957031e-05 nb_pixel_total : 1699 time to create 1 rle with old method : 0.0021157264709472656 length of segment : 73 time for calcul the mask position with numpy : 0.0003490447998046875 nb_pixel_total : 18556 time to create 1 rle with old method : 0.0209963321685791 length of segment : 257 time for calcul the mask position with numpy : 0.0009427070617675781 nb_pixel_total : 61978 time to create 1 rle with old method : 0.06771540641784668 length of segment : 317 time for calcul the mask position with numpy : 0.000484466552734375 nb_pixel_total : 23036 time to create 1 rle with old method : 0.027268409729003906 length of segment : 196 time for calcul the mask position with numpy : 0.0004885196685791016 nb_pixel_total : 24098 time to create 1 rle with old method : 0.028422832489013672 length of segment : 225 time for calcul the mask position with numpy : 0.0002639293670654297 nb_pixel_total : 9587 time to create 1 rle with old method : 0.01114964485168457 length of segment : 137 time for calcul the mask position with numpy : 0.00031566619873046875 nb_pixel_total : 17892 time to create 1 rle with old method : 0.020699024200439453 length of segment : 122 time for calcul the mask position with numpy : 0.00067901611328125 nb_pixel_total : 25239 time to create 1 rle with old method : 0.0280764102935791 length of segment : 271 time for calcul the mask position with numpy : 0.0012443065643310547 nb_pixel_total : 78669 time to create 1 rle with old method : 0.08370113372802734 length of segment : 381 time for calcul the mask position with numpy : 0.0003707408905029297 nb_pixel_total : 19934 time to create 1 rle with old method : 0.021457672119140625 length of segment : 293 time for calcul the mask position with numpy : 0.000225067138671875 nb_pixel_total : 14739 time to create 1 rle with old method : 0.015843868255615234 length of segment : 120 time for calcul the mask position with numpy : 0.0014920234680175781 nb_pixel_total : 114815 time to create 1 rle with old method : 0.11859655380249023 length of segment : 547 time for calcul the mask position with numpy : 0.001069784164428711 nb_pixel_total : 71958 time to create 1 rle with old method : 0.07662844657897949 length of segment : 261 time for calcul the mask position with numpy : 0.00015544891357421875 nb_pixel_total : 6646 time to create 1 rle with old method : 0.00741267204284668 length of segment : 96 time for calcul the mask position with numpy : 0.0008182525634765625 nb_pixel_total : 52190 time to create 1 rle with old method : 0.06443428993225098 length of segment : 370 time for calcul the mask position with numpy : 0.0014641284942626953 nb_pixel_total : 67653 time to create 1 rle with old method : 0.07543468475341797 length of segment : 515 time for calcul the mask position with numpy : 0.00040793418884277344 nb_pixel_total : 22638 time to create 1 rle with old method : 0.024102449417114258 length of segment : 310 time for calcul the mask position with numpy : 0.002518177032470703 nb_pixel_total : 186971 time to create 1 rle with new method : 0.0061037540435791016 length of segment : 530 time for calcul the mask position with numpy : 6.628036499023438e-05 nb_pixel_total : 1590 time to create 1 rle with old method : 0.0019521713256835938 length of segment : 47 time for calcul the mask position with numpy : 0.001958608627319336 nb_pixel_total : 62857 time to create 1 rle with old method : 0.06751012802124023 length of segment : 290 time for calcul the mask position with numpy : 0.0013935565948486328 nb_pixel_total : 31904 time to create 1 rle with old method : 0.03372931480407715 length of segment : 297 time for calcul the mask position with numpy : 0.0006892681121826172 nb_pixel_total : 11991 time to create 1 rle with old method : 0.013488531112670898 length of segment : 116 time for calcul the mask position with numpy : 0.0008752346038818359 nb_pixel_total : 33101 time to create 1 rle with old method : 0.0358586311340332 length of segment : 172 time for calcul the mask position with numpy : 0.0001919269561767578 nb_pixel_total : 3012 time to create 1 rle with old method : 0.00344085693359375 length of segment : 61 time for calcul the mask position with numpy : 0.0002663135528564453 nb_pixel_total : 6040 time to create 1 rle with old method : 0.007457733154296875 length of segment : 76 time for calcul the mask position with numpy : 0.00024271011352539062 nb_pixel_total : 3185 time to create 1 rle with old method : 0.0037817955017089844 length of segment : 69 time for calcul the mask position with numpy : 0.0005133152008056641 nb_pixel_total : 11321 time to create 1 rle with old method : 0.012447357177734375 length of segment : 149 time for calcul the mask position with numpy : 0.005316257476806641 nb_pixel_total : 170550 time to create 1 rle with new method : 0.0071277618408203125 length of segment : 564 time for calcul the mask position with numpy : 0.00513458251953125 nb_pixel_total : 93132 time to create 1 rle with old method : 0.09929060935974121 length of segment : 832 time for calcul the mask position with numpy : 0.0009062290191650391 nb_pixel_total : 19670 time to create 1 rle with old method : 0.021582365036010742 length of segment : 268 time for calcul the mask position with numpy : 0.0008537769317626953 nb_pixel_total : 19792 time to create 1 rle with old method : 0.021525859832763672 length of segment : 224 time for calcul the mask position with numpy : 0.0032727718353271484 nb_pixel_total : 125778 time to create 1 rle with old method : 0.13327932357788086 length of segment : 863 time for calcul the mask position with numpy : 0.0016558170318603516 nb_pixel_total : 43171 time to create 1 rle with old method : 0.04600334167480469 length of segment : 368 time for calcul the mask position with numpy : 0.00013065338134765625 nb_pixel_total : 913 time to create 1 rle with old method : 0.0011811256408691406 length of segment : 36 time for calcul the mask position with numpy : 0.0006473064422607422 nb_pixel_total : 33191 time to create 1 rle with old method : 0.03595161437988281 length of segment : 278 time for calcul the mask position with numpy : 0.0021538734436035156 nb_pixel_total : 28681 time to create 1 rle with old method : 0.03168177604675293 length of segment : 312 time for calcul the mask position with numpy : 0.0007963180541992188 nb_pixel_total : 18181 time to create 1 rle with old method : 0.020490169525146484 length of segment : 140 time for calcul the mask position with numpy : 0.0019919872283935547 nb_pixel_total : 35930 time to create 1 rle with old method : 0.0394291877746582 length of segment : 288 time for calcul the mask position with numpy : 0.0008535385131835938 nb_pixel_total : 24695 time to create 1 rle with old method : 0.0275728702545166 length of segment : 312 time for calcul the mask position with numpy : 0.00017333030700683594 nb_pixel_total : 2087 time to create 1 rle with old method : 0.0024411678314208984 length of segment : 56 time for calcul the mask position with numpy : 0.001291513442993164 nb_pixel_total : 38677 time to create 1 rle with old method : 0.04285597801208496 length of segment : 256 time for calcul the mask position with numpy : 0.0008671283721923828 nb_pixel_total : 29009 time to create 1 rle with old method : 0.03228330612182617 length of segment : 198 time for calcul the mask position with numpy : 0.0005414485931396484 nb_pixel_total : 7336 time to create 1 rle with old method : 0.009007453918457031 length of segment : 80 time for calcul the mask position with numpy : 0.0003345012664794922 nb_pixel_total : 8542 time to create 1 rle with old method : 0.010257720947265625 length of segment : 87 time for calcul the mask position with numpy : 0.002866983413696289 nb_pixel_total : 36838 time to create 1 rle with old method : 0.06640911102294922 length of segment : 216 time for calcul the mask position with numpy : 0.0006232261657714844 nb_pixel_total : 29194 time to create 1 rle with old method : 0.031354665756225586 length of segment : 287 time for calcul the mask position with numpy : 0.009228229522705078 nb_pixel_total : 277028 time to create 1 rle with new method : 0.018437623977661133 length of segment : 1254 time for calcul the mask position with numpy : 0.05205416679382324 nb_pixel_total : 1232274 time to create 1 rle with new method : 1.0702400207519531 length of segment : 2669 time for calcul the mask position with numpy : 0.0014374256134033203 nb_pixel_total : 43014 time to create 1 rle with old method : 0.0491328239440918 length of segment : 532 time for calcul the mask position with numpy : 0.023694753646850586 nb_pixel_total : 1566190 time to create 1 rle with new method : 0.08717799186706543 length of segment : 1718 time for calcul the mask position with numpy : 0.0010590553283691406 nb_pixel_total : 43919 time to create 1 rle with old method : 0.04531359672546387 length of segment : 316 time for calcul the mask position with numpy : 0.0011005401611328125 nb_pixel_total : 52245 time to create 1 rle with old method : 0.07225513458251953 length of segment : 227 time for calcul the mask position with numpy : 0.0011050701141357422 nb_pixel_total : 41759 time to create 1 rle with old method : 0.04439234733581543 length of segment : 180 time for calcul the mask position with numpy : 0.004587411880493164 nb_pixel_total : 298733 time to create 1 rle with new method : 0.007521390914916992 length of segment : 861 time for calcul the mask position with numpy : 0.009193658828735352 nb_pixel_total : 461274 time to create 1 rle with new method : 0.030501604080200195 length of segment : 1037 time for calcul the mask position with numpy : 0.0015501976013183594 nb_pixel_total : 85021 time to create 1 rle with old method : 0.09070277214050293 length of segment : 376 time for calcul the mask position with numpy : 0.004322052001953125 nb_pixel_total : 247966 time to create 1 rle with new method : 0.006087303161621094 length of segment : 577 time for calcul the mask position with numpy : 0.003166675567626953 nb_pixel_total : 264088 time to create 1 rle with new method : 0.00690007209777832 length of segment : 600 time for calcul the mask position with numpy : 0.004643678665161133 nb_pixel_total : 166008 time to create 1 rle with new method : 0.0078084468841552734 length of segment : 750 time for calcul the mask position with numpy : 0.00040030479431152344 nb_pixel_total : 11645 time to create 1 rle with old method : 0.013398885726928711 length of segment : 104 time for calcul the mask position with numpy : 0.012774944305419922 nb_pixel_total : 518069 time to create 1 rle with new method : 0.027402400970458984 length of segment : 1021 time for calcul the mask position with numpy : 0.0020270347595214844 nb_pixel_total : 61932 time to create 1 rle with old method : 0.06847500801086426 length of segment : 422 time for calcul the mask position with numpy : 0.0009799003601074219 nb_pixel_total : 33561 time to create 1 rle with old method : 0.03706812858581543 length of segment : 209 time for calcul the mask position with numpy : 0.00508427619934082 nb_pixel_total : 229069 time to create 1 rle with new method : 0.00744938850402832 length of segment : 664 time for calcul the mask position with numpy : 0.003092050552368164 nb_pixel_total : 108871 time to create 1 rle with old method : 0.11897015571594238 length of segment : 520 time for calcul the mask position with numpy : 0.00030612945556640625 nb_pixel_total : 7778 time to create 1 rle with old method : 0.009103536605834961 length of segment : 148 time for calcul the mask position with numpy : 0.0002837181091308594 nb_pixel_total : 4348 time to create 1 rle with old method : 0.0050182342529296875 length of segment : 112 time for calcul the mask position with numpy : 0.03256964683532715 nb_pixel_total : 833698 time to create 1 rle with new method : 0.9839992523193359 length of segment : 2109 time for calcul the mask position with numpy : 0.001073598861694336 nb_pixel_total : 67608 time to create 1 rle with old method : 0.07129240036010742 length of segment : 413 time for calcul the mask position with numpy : 8.034706115722656e-05 nb_pixel_total : 1226 time to create 1 rle with old method : 0.00142669677734375 length of segment : 45 time for calcul the mask position with numpy : 0.0004329681396484375 nb_pixel_total : 8821 time to create 1 rle with old method : 0.009443521499633789 length of segment : 199 time for calcul the mask position with numpy : 0.002241849899291992 nb_pixel_total : 57607 time to create 1 rle with old method : 0.061139822006225586 length of segment : 437 time for calcul the mask position with numpy : 0.0004055500030517578 nb_pixel_total : 6457 time to create 1 rle with old method : 0.007397651672363281 length of segment : 133 time for calcul the mask position with numpy : 0.0009353160858154297 nb_pixel_total : 11047 time to create 1 rle with old method : 0.012258768081665039 length of segment : 194 time for calcul the mask position with numpy : 0.001233816146850586 nb_pixel_total : 25149 time to create 1 rle with old method : 0.029039621353149414 length of segment : 300 time for calcul the mask position with numpy : 0.0028204917907714844 nb_pixel_total : 87624 time to create 1 rle with old method : 0.09518694877624512 length of segment : 379 time for calcul the mask position with numpy : 0.0010249614715576172 nb_pixel_total : 24665 time to create 1 rle with old method : 0.027124643325805664 length of segment : 172 time for calcul the mask position with numpy : 0.001150369644165039 nb_pixel_total : 21956 time to create 1 rle with old method : 0.024110794067382812 length of segment : 431 time for calcul the mask position with numpy : 0.0010166168212890625 nb_pixel_total : 20218 time to create 1 rle with old method : 0.0222623348236084 length of segment : 430 time for calcul the mask position with numpy : 0.005839109420776367 nb_pixel_total : 160482 time to create 1 rle with new method : 0.008429765701293945 length of segment : 573 time for calcul the mask position with numpy : 0.00015497207641601562 nb_pixel_total : 1620 time to create 1 rle with old method : 0.00197601318359375 length of segment : 52 time for calcul the mask position with numpy : 0.00016355514526367188 nb_pixel_total : 3604 time to create 1 rle with old method : 0.004651546478271484 length of segment : 95 time for calcul the mask position with numpy : 0.0001533031463623047 nb_pixel_total : 6237 time to create 1 rle with old method : 0.007050037384033203 length of segment : 101 time for calcul the mask position with numpy : 0.026181697845458984 nb_pixel_total : 930231 time to create 1 rle with new method : 0.7293305397033691 length of segment : 1269 time for calcul the mask position with numpy : 0.001596689224243164 nb_pixel_total : 48293 time to create 1 rle with old method : 0.05189180374145508 length of segment : 375 time for calcul the mask position with numpy : 0.00043654441833496094 nb_pixel_total : 10207 time to create 1 rle with old method : 0.011366605758666992 length of segment : 108 time for calcul the mask position with numpy : 0.012385368347167969 nb_pixel_total : 420500 time to create 1 rle with new method : 0.0416715145111084 length of segment : 1114 time for calcul the mask position with numpy : 0.0011141300201416016 nb_pixel_total : 69061 time to create 1 rle with old method : 0.07396125793457031 length of segment : 234 time for calcul the mask position with numpy : 0.002644777297973633 nb_pixel_total : 63850 time to create 1 rle with old method : 0.06913113594055176 length of segment : 359 time for calcul the mask position with numpy : 0.003910541534423828 nb_pixel_total : 109418 time to create 1 rle with old method : 0.11850094795227051 length of segment : 536 time for calcul the mask position with numpy : 0.0010378360748291016 nb_pixel_total : 41418 time to create 1 rle with old method : 0.047242164611816406 length of segment : 368 time for calcul the mask position with numpy : 0.006325483322143555 nb_pixel_total : 299592 time to create 1 rle with new method : 0.008835077285766602 length of segment : 532 time for calcul the mask position with numpy : 0.0011451244354248047 nb_pixel_total : 48251 time to create 1 rle with old method : 0.051523447036743164 length of segment : 409 time for calcul the mask position with numpy : 0.0009932518005371094 nb_pixel_total : 40965 time to create 1 rle with old method : 0.0465090274810791 length of segment : 457 time for calcul the mask position with numpy : 0.006874561309814453 nb_pixel_total : 273197 time to create 1 rle with new method : 0.00848078727722168 length of segment : 374 time for calcul the mask position with numpy : 0.0007605552673339844 nb_pixel_total : 10262 time to create 1 rle with old method : 0.01206350326538086 length of segment : 136 time for calcul the mask position with numpy : 0.0009591579437255859 nb_pixel_total : 23951 time to create 1 rle with old method : 0.03891301155090332 length of segment : 231 time for calcul the mask position with numpy : 0.8065352439880371 nb_pixel_total : 3053520 time to create 1 rle with new method : 0.7336666584014893 length of segment : 3410 time for calcul the mask position with numpy : 0.0003762245178222656 nb_pixel_total : 3043 time to create 1 rle with old method : 0.0037467479705810547 length of segment : 104 time for calcul the mask position with numpy : 0.0001933574676513672 nb_pixel_total : 3444 time to create 1 rle with old method : 0.0041196346282958984 length of segment : 86 time for calcul the mask position with numpy : 0.003168821334838867 nb_pixel_total : 123680 time to create 1 rle with old method : 0.13508129119873047 length of segment : 444 time for calcul the mask position with numpy : 0.0005173683166503906 nb_pixel_total : 16151 time to create 1 rle with old method : 0.018979310989379883 length of segment : 158 time for calcul the mask position with numpy : 0.00018286705017089844 nb_pixel_total : 4318 time to create 1 rle with old method : 0.0054323673248291016 length of segment : 87 time for calcul the mask position with numpy : 0.00045228004455566406 nb_pixel_total : 13962 time to create 1 rle with old method : 0.015992403030395508 length of segment : 118 time for calcul the mask position with numpy : 0.0013599395751953125 nb_pixel_total : 28327 time to create 1 rle with old method : 0.03580737113952637 length of segment : 693 time for calcul the mask position with numpy : 0.00047469139099121094 nb_pixel_total : 14344 time to create 1 rle with old method : 0.02377629280090332 length of segment : 114 time for calcul the mask position with numpy : 0.004388093948364258 nb_pixel_total : 116574 time to create 1 rle with old method : 0.13698816299438477 length of segment : 370 time for calcul the mask position with numpy : 0.0008893013000488281 nb_pixel_total : 12420 time to create 1 rle with old method : 0.014009714126586914 length of segment : 262 time for calcul the mask position with numpy : 0.003949642181396484 nb_pixel_total : 124236 time to create 1 rle with old method : 0.1462845802307129 length of segment : 310 time for calcul the mask position with numpy : 0.0051996707916259766 nb_pixel_total : 172334 time to create 1 rle with new method : 0.009605884552001953 length of segment : 518 time for calcul the mask position with numpy : 0.0004119873046875 nb_pixel_total : 6319 time to create 1 rle with old method : 0.007020235061645508 length of segment : 148 time for calcul the mask position with numpy : 0.0001251697540283203 nb_pixel_total : 1981 time to create 1 rle with old method : 0.002345561981201172 length of segment : 60 time for calcul the mask position with numpy : 0.002415180206298828 nb_pixel_total : 53217 time to create 1 rle with old method : 0.058576345443725586 length of segment : 782 time for calcul the mask position with numpy : 0.0008463859558105469 nb_pixel_total : 17866 time to create 1 rle with old method : 0.020799875259399414 length of segment : 280 time for calcul the mask position with numpy : 0.008582830429077148 nb_pixel_total : 231892 time to create 1 rle with new method : 0.013837575912475586 length of segment : 632 time for calcul the mask position with numpy : 0.002262115478515625 nb_pixel_total : 104600 time to create 1 rle with old method : 0.11136174201965332 length of segment : 398 time for calcul the mask position with numpy : 0.013017892837524414 nb_pixel_total : 473165 time to create 1 rle with new method : 0.031436920166015625 length of segment : 824 time for calcul the mask position with numpy : 0.0020284652709960938 nb_pixel_total : 58977 time to create 1 rle with old method : 0.08005499839782715 length of segment : 304 time for calcul the mask position with numpy : 0.0009436607360839844 nb_pixel_total : 25291 time to create 1 rle with old method : 0.028725862503051758 length of segment : 171 time for calcul the mask position with numpy : 0.0006375312805175781 nb_pixel_total : 21649 time to create 1 rle with old method : 0.02412867546081543 length of segment : 147 time for calcul the mask position with numpy : 0.026378393173217773 nb_pixel_total : 899516 time to create 1 rle with new method : 0.22002053260803223 length of segment : 1850 time for calcul the mask position with numpy : 0.00032258033752441406 nb_pixel_total : 12082 time to create 1 rle with old method : 0.014058589935302734 length of segment : 219 time for calcul the mask position with numpy : 0.0002639293670654297 nb_pixel_total : 4300 time to create 1 rle with old method : 0.004977226257324219 length of segment : 142 time for calcul the mask position with numpy : 0.0007939338684082031 nb_pixel_total : 28988 time to create 1 rle with old method : 0.05681443214416504 length of segment : 92 time for calcul the mask position with numpy : 0.010906219482421875 nb_pixel_total : 176420 time to create 1 rle with new method : 0.023448705673217773 length of segment : 824 time for calcul the mask position with numpy : 0.0035936832427978516 nb_pixel_total : 97257 time to create 1 rle with old method : 0.10275030136108398 length of segment : 560 time for calcul the mask position with numpy : 0.0018110275268554688 nb_pixel_total : 60974 time to create 1 rle with old method : 0.06875085830688477 length of segment : 398 time for calcul the mask position with numpy : 0.00083160400390625 nb_pixel_total : 28668 time to create 1 rle with old method : 0.03244972229003906 length of segment : 202 time for calcul the mask position with numpy : 0.0008208751678466797 nb_pixel_total : 13416 time to create 1 rle with old method : 0.015239238739013672 length of segment : 221 time for calcul the mask position with numpy : 0.001630544662475586 nb_pixel_total : 44011 time to create 1 rle with old method : 0.04833793640136719 length of segment : 502 time for calcul the mask position with numpy : 0.012531518936157227 nb_pixel_total : 390431 time to create 1 rle with new method : 0.20456171035766602 length of segment : 1091 time for calcul the mask position with numpy : 0.0010843276977539062 nb_pixel_total : 37965 time to create 1 rle with old method : 0.041426897048950195 length of segment : 287 time for calcul the mask position with numpy : 0.00023627281188964844 nb_pixel_total : 6570 time to create 1 rle with old method : 0.0076160430908203125 length of segment : 112 time for calcul the mask position with numpy : 0.001901865005493164 nb_pixel_total : 81050 time to create 1 rle with old method : 0.08695769309997559 length of segment : 361 time for calcul the mask position with numpy : 0.0006704330444335938 nb_pixel_total : 22449 time to create 1 rle with old method : 0.02482008934020996 length of segment : 181 time for calcul the mask position with numpy : 0.0003380775451660156 nb_pixel_total : 10887 time to create 1 rle with old method : 0.012773990631103516 length of segment : 101 time for calcul the mask position with numpy : 0.002555370330810547 nb_pixel_total : 89870 time to create 1 rle with old method : 0.09433698654174805 length of segment : 452 time for calcul the mask position with numpy : 0.00932621955871582 nb_pixel_total : 429466 time to create 1 rle with new method : 0.016638755798339844 length of segment : 1109 time for calcul the mask position with numpy : 0.0020482540130615234 nb_pixel_total : 166800 time to create 1 rle with new method : 0.004535675048828125 length of segment : 516 time for calcul the mask position with numpy : 0.0001850128173828125 nb_pixel_total : 1706 time to create 1 rle with old method : 0.0029861927032470703 length of segment : 34 time for calcul the mask position with numpy : 0.00036144256591796875 nb_pixel_total : 11978 time to create 1 rle with old method : 0.019718170166015625 length of segment : 126 time for calcul the mask position with numpy : 0.0002505779266357422 nb_pixel_total : 8276 time to create 1 rle with old method : 0.013921022415161133 length of segment : 102 time for calcul the mask position with numpy : 0.000133514404296875 nb_pixel_total : 3789 time to create 1 rle with old method : 0.004303693771362305 length of segment : 127 time spent for convertir_results : 25.840182065963745 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 169 chid ids of type : 3663 Number RLEs to save : 69687 save missing photos in datou_result : time spend for datou_step_exec : 72.90015172958374 time spend to save output : 4.707383871078491 total time spend for step 1 : 77.60753560066223 step2:crop_condition Tue Feb 11 00:05:48 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 169 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 12 About to insert : list_path_to_insert length 12 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 14 About to insert : list_path_to_insert length 14 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 12 About to insert : list_path_to_insert length 12 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 44 About to insert : list_path_to_insert length 44 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1336616382, 1336616380, 1336616377, 1336616376, 1336616375, 1336616121, 1336562176, 1336562174, 1336562169, 1336554334] Looping around the photos to save general results len do output : 94 /-3668736870Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736869Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736882Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736888Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736925Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736924Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736922Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736931Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736954Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668737001Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668737003Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668737005Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736881Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736898Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736899Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736909Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736940Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736951Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736963Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736986Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736971Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736983Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668737020Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668737019Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668737012Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668737029Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736872Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736880Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736883Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736890Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736893Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736930Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736932Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736936Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736984Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736988Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668737018Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668737033Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736873Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736886Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736887Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736876Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736875Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736877Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736896Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736897Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736907Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736904Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736903Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736912Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736910Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736911Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736923Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736939Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736927Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736914Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736938Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736929Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736935Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736915Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736941Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736950Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736946Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736948Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736957Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736958Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736982Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736981Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736974Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736979Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736987Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736972Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736989Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736995Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668737010Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668737004Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668737022Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668737014Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668737009Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668737021Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668737013Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668737031Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736892Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736919Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736933Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736968Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736998Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736884Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736906Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736959Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668736956Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668737016Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668737028Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3668737000Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616382', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616380', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616377', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616376', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616375', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616121', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336562176', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336562174', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336562169', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336554334', None, None, None, None, None, '2574201') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 292 time used for this insertion : 0.03912496566772461 save_final save missing photos in datou_result : time spend for datou_step_exec : 44.58957481384277 time spend to save output : 0.04145407676696777 total time spend for step 2 : 44.63102889060974 step3:thcl Tue Feb 11 00:06:32 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.015205621719360352 time to convert the images to numpy array : 0.14027023315429688 time to import caffe and check if the image exist : 0.018104076385498047 time to convert the images to numpy array : 0.19822049140930176 time to import caffe and check if the image exist : 0.012706756591796875 time to convert the images to numpy array : 0.21712970733642578 time to import caffe and check if the image exist : 0.01839137077331543 time to convert the images to numpy array : 0.21377158164978027 time to import caffe and check if the image exist : 0.02131342887878418 time to convert the images to numpy array : 0.22149419784545898 time to import caffe and check if the image exist : 0.021580219268798828 time to convert the images to numpy array : 0.5844602584838867 time to import caffe and check if the image exist : 0.019312143325805664 time to convert the images to numpy array : 0.5997872352600098 time to import caffe and check if the image exist : 0.014384984970092773 time to convert the images to numpy array : 0.6090328693389893 time to import caffe and check if the image exist : 0.024328947067260742 time to convert the images to numpy array : 0.6136922836303711 time to import caffe and check if the image exist : 0.02182292938232422 time to convert the images to numpy array : 0.6708035469055176 total time to convert the images to numpy array : 1.0941574573516846 list photo_ids error: [] list photo_ids correct : [-3668736956, -3668737016, -3668737028, -3668737000, -3668736910, -3668736911, -3668736923, -3668736939, -3668736927, -3668736914, -3668736938, -3668736929, -3668736935, -3668736915, -3668737013, -3668737031, -3668736892, -3668736919, -3668736933, -3668736968, -3668736998, -3668736884, -3668736906, -3668736959, -3668736870, -3668736869, -3668736882, -3668736888, -3668736925, -3668736924, -3668736922, -3668736931, -3668736954, -3668737001, -3668736893, -3668736930, -3668736932, -3668736936, -3668736984, -3668736988, -3668737018, -3668737033, -3668736873, -3668736886, -3668736887, -3668736876, -3668736875, -3668736877, -3668736896, -3668736897, -3668736907, -3668736904, -3668736903, -3668736912, -3668737003, -3668737005, -3668736881, -3668736898, -3668736899, -3668736909, -3668736940, -3668736951, -3668736963, -3668736986, -3668736987, -3668736972, -3668736989, -3668736995, -3668737010, -3668737004, -3668737022, -3668737014, -3668737009, -3668737021, -3668736971, -3668736983, -3668737020, -3668737019, -3668737012, -3668737029, -3668736872, -3668736880, -3668736883, -3668736890, -3668736941, -3668736950, -3668736946, -3668736948, -3668736957, -3668736958, -3668736982, -3668736981, -3668736974, -3668736979] number of photos to traite : 94 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3805 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3586 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.998321533203125 time used to do the prediction : 0.25717592239379883 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3584 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3584 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.1621983051300049 time used to do the prediction : 0.17990446090698242 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 1.8358230590820312e-05 save missing photos in datou_result : time spend for datou_step_exec : 10.927987813949585 time spend to save output : 0.0007762908935546875 total time spend for step 3 : 10.92876410484314 step4:merge_mask_thcl_custom Tue Feb 11 00:06:43 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 169 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1336616382, 1336616380, 1336616377, 1336616376, 1336616375, 1336616121, 1336562176, 1336562174, 1336562169, 1336554334] Looping around the photos to save general results len do output : 10 /1336616382Didn't retrieve data .Didn't retrieve data . /1336616380Didn't retrieve data .Didn't retrieve data . /1336616377Didn't retrieve data .Didn't retrieve data . /1336616376Didn't retrieve data .Didn't retrieve data . /1336616375Didn't retrieve data .Didn't retrieve data . /1336616121Didn't retrieve data .Didn't retrieve data . /1336562176Didn't retrieve data .Didn't retrieve data . /1336562174Didn't retrieve data .Didn't retrieve data . /1336562169Didn't retrieve data .Didn't retrieve data . /1336554334Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616382', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616380', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616377', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616376', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616375', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616121', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336562176', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336562174', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336562169', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336554334', None, None, None, None, None, '2574201') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013909578323364258 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.06274199485778809 time spend to save output : 0.014419317245483398 total time spend for step 4 : 0.07716131210327148 step5:rle_unique_nms_with_priority Tue Feb 11 00:06:43 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 19 nb_hashtags : 6 time to prepare the origin masks : 18.41182589530945 time for calcul the mask position with numpy : 0.4626288414001465 nb_pixel_total : 4462514 time to create 1 rle with new method : 1.2576894760131836 time for calcul the mask position with numpy : 0.021564006805419922 nb_pixel_total : 95348 time to create 1 rle with old method : 0.09754252433776855 time for calcul the mask position with numpy : 0.019495487213134766 nb_pixel_total : 8672 time to create 1 rle with old method : 0.009217500686645508 time for calcul the mask position with numpy : 0.02037668228149414 nb_pixel_total : 36614 time to create 1 rle with old method : 0.042110443115234375 time for calcul the mask position with numpy : 0.019718408584594727 nb_pixel_total : 7801 time to create 1 rle with old method : 0.009171247482299805 time for calcul the mask position with numpy : 0.020332813262939453 nb_pixel_total : 107342 time to create 1 rle with old method : 0.10606169700622559 time for calcul the mask position with numpy : 0.019669532775878906 nb_pixel_total : 41973 time to create 1 rle with old method : 0.041166067123413086 time for calcul the mask position with numpy : 0.020172119140625 nb_pixel_total : 40250 time to create 1 rle with old method : 0.04395651817321777 time for calcul the mask position with numpy : 0.02395462989807129 nb_pixel_total : 427456 time to create 1 rle with new method : 0.5174853801727295 time for calcul the mask position with numpy : 0.02076101303100586 nb_pixel_total : 4814 time to create 1 rle with old method : 0.005173444747924805 time for calcul the mask position with numpy : 0.026724815368652344 nb_pixel_total : 1189 time to create 1 rle with old method : 0.0013647079467773438 time for calcul the mask position with numpy : 0.032534122467041016 nb_pixel_total : 17510 time to create 1 rle with old method : 0.018204450607299805 time for calcul the mask position with numpy : 0.03374743461608887 nb_pixel_total : 26335 time to create 1 rle with old method : 0.028921127319335938 time for calcul the mask position with numpy : 0.04795050621032715 nb_pixel_total : 1407803 time to create 1 rle with new method : 1.0080044269561768 time for calcul the mask position with numpy : 0.031973838806152344 nb_pixel_total : 95436 time to create 1 rle with old method : 0.09470820426940918 time for calcul the mask position with numpy : 0.01981186866760254 nb_pixel_total : 188 time to create 1 rle with old method : 0.00026226043701171875 time for calcul the mask position with numpy : 0.020956039428710938 nb_pixel_total : 107711 time to create 1 rle with old method : 0.10805606842041016 time for calcul the mask position with numpy : 0.020188331604003906 nb_pixel_total : 53721 time to create 1 rle with old method : 0.05308842658996582 time for calcul the mask position with numpy : 0.0202023983001709 nb_pixel_total : 66509 time to create 1 rle with old method : 0.06628084182739258 time for calcul the mask position with numpy : 0.02022576332092285 nb_pixel_total : 41054 time to create 1 rle with old method : 0.04401755332946777 create new chi : 4.552741765975952 time to delete rle : 0.01006937026977539 batch 1 Loaded 20 chid ids of type : 3726 Number RLEs to save : 16418 TO DO : save crop sub photo not yet done ! save time : 1.4033427238464355 nb_obj : 12 nb_hashtags : 5 time to prepare the origin masks : 17.07169771194458 time for calcul the mask position with numpy : 0.7536864280700684 nb_pixel_total : 6443096 time to create 1 rle with new method : 1.1226134300231934 time for calcul the mask position with numpy : 0.02217268943786621 nb_pixel_total : 248793 time to create 1 rle with new method : 0.8833961486816406 time for calcul the mask position with numpy : 0.0211789608001709 nb_pixel_total : 61819 time to create 1 rle with old method : 0.0643775463104248 time for calcul the mask position with numpy : 0.02080368995666504 nb_pixel_total : 17554 time to create 1 rle with old method : 0.019155502319335938 time for calcul the mask position with numpy : 0.020812511444091797 nb_pixel_total : 24029 time to create 1 rle with old method : 0.025736331939697266 time for calcul the mask position with numpy : 0.02130436897277832 nb_pixel_total : 22906 time to create 1 rle with old method : 0.024237871170043945 time for calcul the mask position with numpy : 0.0206148624420166 nb_pixel_total : 19925 time to create 1 rle with old method : 0.021647930145263672 time for calcul the mask position with numpy : 0.02021169662475586 nb_pixel_total : 1688 time to create 1 rle with old method : 0.001756906509399414 time for calcul the mask position with numpy : 0.020583391189575195 nb_pixel_total : 1738 time to create 1 rle with old method : 0.0019333362579345703 time for calcul the mask position with numpy : 0.021529436111450195 nb_pixel_total : 38454 time to create 1 rle with old method : 0.03969526290893555 time for calcul the mask position with numpy : 0.02073216438293457 nb_pixel_total : 18489 time to create 1 rle with old method : 0.020945310592651367 time for calcul the mask position with numpy : 0.022078275680541992 nb_pixel_total : 147156 time to create 1 rle with old method : 0.16066789627075195 time for calcul the mask position with numpy : 0.025345325469970703 nb_pixel_total : 4593 time to create 1 rle with old method : 0.005912065505981445 create new chi : 3.4692904949188232 time to delete rle : 0.0008580684661865234 batch 1 Loaded 13 chid ids of type : 3726 Number RLEs to save : 8311 TO DO : save crop sub photo not yet done ! save time : 0.8351833820343018 nb_obj : 13 nb_hashtags : 5 time to prepare the origin masks : 14.632139205932617 time for calcul the mask position with numpy : 0.7491223812103271 nb_pixel_total : 6383726 time to create 1 rle with new method : 0.911517858505249 time for calcul the mask position with numpy : 0.021161317825317383 nb_pixel_total : 70889 time to create 1 rle with old method : 0.07475733757019043 time for calcul the mask position with numpy : 0.03702235221862793 nb_pixel_total : 186880 time to create 1 rle with new method : 0.9380955696105957 time for calcul the mask position with numpy : 0.038843631744384766 nb_pixel_total : 1582 time to create 1 rle with old method : 0.002556324005126953 time for calcul the mask position with numpy : 0.028563737869262695 nb_pixel_total : 114976 time to create 1 rle with old method : 0.12462759017944336 time for calcul the mask position with numpy : 0.020926475524902344 nb_pixel_total : 71855 time to create 1 rle with old method : 0.07456064224243164 time for calcul the mask position with numpy : 0.02112865447998047 nb_pixel_total : 78551 time to create 1 rle with old method : 0.08183598518371582 time for calcul the mask position with numpy : 0.02105855941772461 nb_pixel_total : 6604 time to create 1 rle with old method : 0.007615327835083008 time for calcul the mask position with numpy : 0.0214080810546875 nb_pixel_total : 19845 time to create 1 rle with old method : 0.020786523818969727 time for calcul the mask position with numpy : 0.021105051040649414 nb_pixel_total : 14648 time to create 1 rle with old method : 0.01526498794555664 time for calcul the mask position with numpy : 0.02168560028076172 nb_pixel_total : 52135 time to create 1 rle with old method : 0.0541379451751709 time for calcul the mask position with numpy : 0.021513938903808594 nb_pixel_total : 8111 time to create 1 rle with old method : 0.00905919075012207 time for calcul the mask position with numpy : 0.02144026756286621 nb_pixel_total : 17779 time to create 1 rle with old method : 0.018506765365600586 time for calcul the mask position with numpy : 0.02037358283996582 nb_pixel_total : 22659 time to create 1 rle with old method : 0.02368640899658203 create new chi : 3.493237257003784 time to delete rle : 0.0007948875427246094 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 9603 TO DO : save crop sub photo not yet done ! save time : 0.6644563674926758 nb_obj : 26 nb_hashtags : 6 time to prepare the origin masks : 8.4973304271698 time for calcul the mask position with numpy : 0.8069312572479248 nb_pixel_total : 6288283 time to create 1 rle with new method : 1.2984910011291504 time for calcul the mask position with numpy : 0.028171300888061523 nb_pixel_total : 2081 time to create 1 rle with old method : 0.0023233890533447266 time for calcul the mask position with numpy : 0.028048038482666016 nb_pixel_total : 125592 time to create 1 rle with old method : 0.13022518157958984 time for calcul the mask position with numpy : 0.02757406234741211 nb_pixel_total : 15243 time to create 1 rle with old method : 0.01594686508178711 time for calcul the mask position with numpy : 0.02771925926208496 nb_pixel_total : 11152 time to create 1 rle with old method : 0.012453079223632812 time for calcul the mask position with numpy : 0.028385639190673828 nb_pixel_total : 19616 time to create 1 rle with old method : 0.02089381217956543 time for calcul the mask position with numpy : 0.02805042266845703 nb_pixel_total : 7347 time to create 1 rle with old method : 0.008032083511352539 time for calcul the mask position with numpy : 0.027891159057617188 nb_pixel_total : 38548 time to create 1 rle with old method : 0.04056119918823242 time for calcul the mask position with numpy : 0.027957677841186523 nb_pixel_total : 28920 time to create 1 rle with old method : 0.0301513671875 time for calcul the mask position with numpy : 0.028295516967773438 nb_pixel_total : 35860 time to create 1 rle with old method : 0.03789520263671875 time for calcul the mask position with numpy : 0.026600122451782227 nb_pixel_total : 19740 time to create 1 rle with old method : 0.01982283592224121 time for calcul the mask position with numpy : 0.02659893035888672 nb_pixel_total : 908 time to create 1 rle with old method : 0.0009107589721679688 time for calcul the mask position with numpy : 0.02594733238220215 nb_pixel_total : 4149 time to create 1 rle with old method : 0.004613399505615234 time for calcul the mask position with numpy : 0.027297019958496094 nb_pixel_total : 24548 time to create 1 rle with old method : 0.024706602096557617 time for calcul the mask position with numpy : 0.026746034622192383 nb_pixel_total : 2986 time to create 1 rle with old method : 0.0032439231872558594 time for calcul the mask position with numpy : 0.02636432647705078 nb_pixel_total : 18065 time to create 1 rle with old method : 0.01819920539855957 time for calcul the mask position with numpy : 0.026638507843017578 nb_pixel_total : 8428 time to create 1 rle with old method : 0.009223461151123047 time for calcul the mask position with numpy : 0.026955842971801758 nb_pixel_total : 5969 time to create 1 rle with old method : 0.006267547607421875 time for calcul the mask position with numpy : 0.02792525291442871 nb_pixel_total : 43132 time to create 1 rle with old method : 0.044793128967285156 time for calcul the mask position with numpy : 0.028192758560180664 nb_pixel_total : 31803 time to create 1 rle with old method : 0.03400921821594238 time for calcul the mask position with numpy : 0.02889275550842285 nb_pixel_total : 170393 time to create 1 rle with new method : 1.0942158699035645 time for calcul the mask position with numpy : 0.027821779251098633 nb_pixel_total : 3135 time to create 1 rle with old method : 0.0036478042602539062 time for calcul the mask position with numpy : 0.028150320053100586 nb_pixel_total : 62607 time to create 1 rle with old method : 0.07125711441040039 time for calcul the mask position with numpy : 0.03176140785217285 nb_pixel_total : 29944 time to create 1 rle with old method : 0.03458857536315918 time for calcul the mask position with numpy : 0.02915787696838379 nb_pixel_total : 6973 time to create 1 rle with old method : 0.008285760879516602 time for calcul the mask position with numpy : 0.038789987564086914 nb_pixel_total : 11895 time to create 1 rle with old method : 0.013030767440795898 time for calcul the mask position with numpy : 0.029836177825927734 nb_pixel_total : 32923 time to create 1 rle with old method : 0.035700082778930664 create new chi : 4.6245644092559814 time to delete rle : 0.0014224052429199219 batch 1 Loaded 27 chid ids of type : 3726 Number RLEs to save : 14281 TO DO : save crop sub photo not yet done ! save time : 0.9186604022979736 nb_obj : 11 nb_hashtags : 5 time to prepare the origin masks : 19.593256950378418 time for calcul the mask position with numpy : 0.810835599899292 nb_pixel_total : 3527411 time to create 1 rle with new method : 0.7620067596435547 time for calcul the mask position with numpy : 0.01987433433532715 nb_pixel_total : 4619 time to create 1 rle with old method : 0.00536799430847168 time for calcul the mask position with numpy : 0.01971888542175293 nb_pixel_total : 2572 time to create 1 rle with old method : 0.0029685497283935547 time for calcul the mask position with numpy : 0.021178722381591797 nb_pixel_total : 110514 time to create 1 rle with old method : 0.11233162879943848 time for calcul the mask position with numpy : 0.021521329879760742 nb_pixel_total : 282746 time to create 1 rle with new method : 1.4203598499298096 time for calcul the mask position with numpy : 0.020949840545654297 nb_pixel_total : 263459 time to create 1 rle with new method : 1.026360273361206 time for calcul the mask position with numpy : 0.023595094680786133 nb_pixel_total : 44089 time to create 1 rle with old method : 0.04874563217163086 time for calcul the mask position with numpy : 0.022257328033447266 nb_pixel_total : 27393 time to create 1 rle with old method : 0.029998302459716797 time for calcul the mask position with numpy : 0.023041963577270508 nb_pixel_total : 84082 time to create 1 rle with old method : 0.09070181846618652 time for calcul the mask position with numpy : 0.041008710861206055 nb_pixel_total : 1428668 time to create 1 rle with new method : 0.7893202304840088 time for calcul the mask position with numpy : 0.021170616149902344 nb_pixel_total : 43582 time to create 1 rle with old method : 0.0501863956451416 time for calcul the mask position with numpy : 0.04133296012878418 nb_pixel_total : 1231105 time to create 1 rle with new method : 0.7986223697662354 create new chi : 6.348909616470337 time to delete rle : 0.0018916130065917969 batch 1 Loaded 12 chid ids of type : 3726 Number RLEs to save : 18182 TO DO : save crop sub photo not yet done ! save time : 3.1604838371276855 nb_obj : 13 nb_hashtags : 4 time to prepare the origin masks : 16.941755056381226 time for calcul the mask position with numpy : 0.4539792537689209 nb_pixel_total : 5190197 time to create 1 rle with new method : 0.805487871170044 time for calcul the mask position with numpy : 0.020679950714111328 nb_pixel_total : 187574 time to create 1 rle with new method : 0.5108671188354492 time for calcul the mask position with numpy : 0.021192550659179688 nb_pixel_total : 33411 time to create 1 rle with old method : 0.035550832748413086 time for calcul the mask position with numpy : 0.022427082061767578 nb_pixel_total : 87827 time to create 1 rle with old method : 0.09210491180419922 time for calcul the mask position with numpy : 0.02094244956970215 nb_pixel_total : 8812 time to create 1 rle with old method : 0.009976625442504883 time for calcul the mask position with numpy : 0.021785736083984375 nb_pixel_total : 1218 time to create 1 rle with old method : 0.0014295578002929688 time for calcul the mask position with numpy : 0.021268844604492188 nb_pixel_total : 67291 time to create 1 rle with old method : 0.07161092758178711 time for calcul the mask position with numpy : 0.027293682098388672 nb_pixel_total : 705410 time to create 1 rle with new method : 0.6658799648284912 time for calcul the mask position with numpy : 0.021664142608642578 nb_pixel_total : 4335 time to create 1 rle with old method : 0.0045604705810546875 time for calcul the mask position with numpy : 0.02078866958618164 nb_pixel_total : 7736 time to create 1 rle with old method : 0.008707761764526367 time for calcul the mask position with numpy : 0.021289825439453125 nb_pixel_total : 61858 time to create 1 rle with old method : 0.06454968452453613 time for calcul the mask position with numpy : 0.02130436897277832 nb_pixel_total : 11615 time to create 1 rle with old method : 0.012932300567626953 time for calcul the mask position with numpy : 0.024680614471435547 nb_pixel_total : 517157 time to create 1 rle with new method : 0.7225258350372314 time for calcul the mask position with numpy : 0.022065162658691406 nb_pixel_total : 165799 time to create 1 rle with new method : 0.7565484046936035 create new chi : 4.636410713195801 time to delete rle : 0.001354217529296875 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 14525 TO DO : save crop sub photo not yet done ! save time : 0.907167911529541 nb_obj : 22 nb_hashtags : 6 time to prepare the origin masks : 28.963099241256714 time for calcul the mask position with numpy : 0.6261765956878662 nb_pixel_total : 5013313 time to create 1 rle with new method : 0.9548208713531494 time for calcul the mask position with numpy : 0.03436994552612305 nb_pixel_total : 11001 time to create 1 rle with old method : 0.011957883834838867 time for calcul the mask position with numpy : 0.033408403396606445 nb_pixel_total : 24578 time to create 1 rle with old method : 0.026796817779541016 time for calcul the mask position with numpy : 0.03358793258666992 nb_pixel_total : 16776 time to create 1 rle with old method : 0.017899274826049805 time for calcul the mask position with numpy : 0.037735700607299805 nb_pixel_total : 32172 time to create 1 rle with old method : 0.03266406059265137 time for calcul the mask position with numpy : 0.033036231994628906 nb_pixel_total : 38750 time to create 1 rle with old method : 0.0397648811340332 time for calcul the mask position with numpy : 0.04315519332885742 nb_pixel_total : 927798 time to create 1 rle with new method : 1.21183443069458 time for calcul the mask position with numpy : 0.03351187705993652 nb_pixel_total : 63657 time to create 1 rle with old method : 0.06533622741699219 time for calcul the mask position with numpy : 0.03284096717834473 nb_pixel_total : 68853 time to create 1 rle with old method : 0.06889748573303223 time for calcul the mask position with numpy : 0.03210163116455078 nb_pixel_total : 256 time to create 1 rle with old method : 0.0003161430358886719 time for calcul the mask position with numpy : 0.03464865684509277 nb_pixel_total : 272862 time to create 1 rle with new method : 0.8501338958740234 time for calcul the mask position with numpy : 0.036423444747924805 nb_pixel_total : 3007 time to create 1 rle with old method : 0.003328084945678711 time for calcul the mask position with numpy : 0.03275346755981445 nb_pixel_total : 1613 time to create 1 rle with old method : 0.0017673969268798828 time for calcul the mask position with numpy : 0.03290963172912598 nb_pixel_total : 20104 time to create 1 rle with old method : 0.021141767501831055 time for calcul the mask position with numpy : 0.032758474349975586 nb_pixel_total : 87605 time to create 1 rle with old method : 0.08826041221618652 time for calcul the mask position with numpy : 0.03225207328796387 nb_pixel_total : 25010 time to create 1 rle with old method : 0.025878429412841797 time for calcul the mask position with numpy : 0.032288312911987305 nb_pixel_total : 6435 time to create 1 rle with old method : 0.006876230239868164 time for calcul the mask position with numpy : 0.03278541564941406 nb_pixel_total : 56950 time to create 1 rle with old method : 0.059358835220336914 time for calcul the mask position with numpy : 0.032881975173950195 nb_pixel_total : 48172 time to create 1 rle with old method : 0.06952428817749023 time for calcul the mask position with numpy : 0.04049539566040039 nb_pixel_total : 21769 time to create 1 rle with old method : 0.023371458053588867 time for calcul the mask position with numpy : 0.03509259223937988 nb_pixel_total : 159655 time to create 1 rle with new method : 0.9188096523284912 time for calcul the mask position with numpy : 0.03730607032775879 nb_pixel_total : 109083 time to create 1 rle with old method : 0.1123661994934082 time for calcul the mask position with numpy : 0.0331273078918457 nb_pixel_total : 40821 time to create 1 rle with old method : 0.04310441017150879 create new chi : 6.146234035491943 time to delete rle : 0.0020830631256103516 batch 1 Loaded 23 chid ids of type : 3726 Number RLEs to save : 17694 TO DO : save crop sub photo not yet done ! save time : 1.2626750469207764 nb_obj : 9 nb_hashtags : 3 time to prepare the origin masks : 14.63728141784668 time for calcul the mask position with numpy : 0.23961973190307617 nb_pixel_total : 3812548 time to create 1 rle with new method : 1.3228368759155273 time for calcul the mask position with numpy : 0.03446626663208008 nb_pixel_total : 123311 time to create 1 rle with old method : 0.15875840187072754 time for calcul the mask position with numpy : 0.034181833267211914 nb_pixel_total : 581 time to create 1 rle with old method : 0.0008695125579833984 time for calcul the mask position with numpy : 0.03368544578552246 nb_pixel_total : 13863 time to create 1 rle with old method : 0.014625072479248047 time for calcul the mask position with numpy : 0.03399968147277832 nb_pixel_total : 1955 time to create 1 rle with old method : 0.00231170654296875 time for calcul the mask position with numpy : 0.020674943923950195 nb_pixel_total : 16083 time to create 1 rle with old method : 0.01743149757385254 time for calcul the mask position with numpy : 0.021274328231811523 nb_pixel_total : 3421 time to create 1 rle with old method : 0.0040111541748046875 time for calcul the mask position with numpy : 0.02214789390563965 nb_pixel_total : 3031 time to create 1 rle with old method : 0.0039081573486328125 time for calcul the mask position with numpy : 0.31868886947631836 nb_pixel_total : 3051558 time to create 1 rle with new method : 1.9861030578613281 time for calcul the mask position with numpy : 0.03442788124084473 nb_pixel_total : 23889 time to create 1 rle with old method : 0.02628636360168457 create new chi : 4.397470235824585 time to delete rle : 0.0010755062103271484 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 11473 TO DO : save crop sub photo not yet done ! save time : 0.7632951736450195 nb_obj : 22 nb_hashtags : 6 time to prepare the origin masks : 27.35786533355713 time for calcul the mask position with numpy : 0.4306492805480957 nb_pixel_total : 4385421 time to create 1 rle with new method : 1.0234434604644775 time for calcul the mask position with numpy : 0.03199052810668945 nb_pixel_total : 4056 time to create 1 rle with old method : 0.00435638427734375 time for calcul the mask position with numpy : 0.03374934196472168 nb_pixel_total : 97114 time to create 1 rle with old method : 0.09621071815490723 time for calcul the mask position with numpy : 0.03240084648132324 nb_pixel_total : 28910 time to create 1 rle with old method : 0.03016042709350586 time for calcul the mask position with numpy : 0.03289532661437988 nb_pixel_total : 25169 time to create 1 rle with old method : 0.025331735610961914 time for calcul the mask position with numpy : 0.033463478088378906 nb_pixel_total : 123848 time to create 1 rle with old method : 0.12225532531738281 time for calcul the mask position with numpy : 0.03139853477478027 nb_pixel_total : 28621 time to create 1 rle with old method : 0.028274059295654297 time for calcul the mask position with numpy : 0.030621051788330078 nb_pixel_total : 12060 time to create 1 rle with old method : 0.012568473815917969 time for calcul the mask position with numpy : 0.025467395782470703 nb_pixel_total : 60787 time to create 1 rle with old method : 0.05983781814575195 time for calcul the mask position with numpy : 0.01963067054748535 nb_pixel_total : 6313 time to create 1 rle with old method : 0.006742238998413086 time for calcul the mask position with numpy : 0.020479679107666016 nb_pixel_total : 4288 time to create 1 rle with old method : 0.004696369171142578 time for calcul the mask position with numpy : 0.021990299224853516 nb_pixel_total : 230898 time to create 1 rle with new method : 1.119572401046753 time for calcul the mask position with numpy : 0.03271627426147461 nb_pixel_total : 894974 time to create 1 rle with new method : 1.1422450542449951 time for calcul the mask position with numpy : 0.033806562423706055 nb_pixel_total : 21583 time to create 1 rle with old method : 0.02276301383972168 time for calcul the mask position with numpy : 0.03904080390930176 nb_pixel_total : 471237 time to create 1 rle with new method : 1.4884026050567627 time for calcul the mask position with numpy : 0.033600807189941406 nb_pixel_total : 17800 time to create 1 rle with old method : 0.018236398696899414 time for calcul the mask position with numpy : 0.03374218940734863 nb_pixel_total : 172268 time to create 1 rle with new method : 1.905940294265747 time for calcul the mask position with numpy : 0.048084259033203125 nb_pixel_total : 53072 time to create 1 rle with old method : 0.07049393653869629 time for calcul the mask position with numpy : 0.03935861587524414 nb_pixel_total : 1971 time to create 1 rle with old method : 0.002670764923095703 time for calcul the mask position with numpy : 0.04690980911254883 nb_pixel_total : 12401 time to create 1 rle with old method : 0.014933109283447266 time for calcul the mask position with numpy : 0.03750920295715332 nb_pixel_total : 116383 time to create 1 rle with old method : 0.13215184211730957 time for calcul the mask position with numpy : 0.04458141326904297 nb_pixel_total : 104509 time to create 1 rle with old method : 0.11315083503723145 time for calcul the mask position with numpy : 0.0364687442779541 nb_pixel_total : 176557 time to create 1 rle with new method : 1.0190882682800293 create new chi : 9.796384572982788 time to delete rle : 0.003667116165161133 batch 1 Loaded 23 chid ids of type : 3726 Number RLEs to save : 20529 TO DO : save crop sub photo not yet done ! save time : 1.891474962234497 nb_obj : 14 nb_hashtags : 4 time to prepare the origin masks : 12.112358808517456 time for calcul the mask position with numpy : 1.1949479579925537 nb_pixel_total : 5695036 time to create 1 rle with new method : 1.115825891494751 time for calcul the mask position with numpy : 0.03315901756286621 nb_pixel_total : 3409 time to create 1 rle with old method : 0.0039010047912597656 time for calcul the mask position with numpy : 0.033287763595581055 nb_pixel_total : 380 time to create 1 rle with old method : 0.0005373954772949219 time for calcul the mask position with numpy : 0.03389930725097656 nb_pixel_total : 22326 time to create 1 rle with old method : 0.02361464500427246 time for calcul the mask position with numpy : 0.03440380096435547 nb_pixel_total : 1704 time to create 1 rle with old method : 0.002001047134399414 time for calcul the mask position with numpy : 0.03402876853942871 nb_pixel_total : 92748 time to create 1 rle with old method : 0.09771609306335449 time for calcul the mask position with numpy : 0.04029512405395508 nb_pixel_total : 482427 time to create 1 rle with new method : 0.7663137912750244 time for calcul the mask position with numpy : 0.03518509864807129 nb_pixel_total : 10822 time to create 1 rle with old method : 0.011667251586914062 time for calcul the mask position with numpy : 0.02550792694091797 nb_pixel_total : 80835 time to create 1 rle with old method : 0.08376550674438477 time for calcul the mask position with numpy : 0.02369093894958496 nb_pixel_total : 166719 time to create 1 rle with new method : 0.5298564434051514 time for calcul the mask position with numpy : 0.03551054000854492 nb_pixel_total : 6556 time to create 1 rle with old method : 0.007196903228759766 time for calcul the mask position with numpy : 0.032585859298706055 nb_pixel_total : 37688 time to create 1 rle with old method : 0.04267406463623047 time for calcul the mask position with numpy : 0.03481435775756836 nb_pixel_total : 390381 time to create 1 rle with new method : 0.9217653274536133 time for calcul the mask position with numpy : 0.03113245964050293 nb_pixel_total : 45867 time to create 1 rle with old method : 0.048349857330322266 time for calcul the mask position with numpy : 0.029770612716674805 nb_pixel_total : 13342 time to create 1 rle with old method : 0.01442718505859375 create new chi : 5.436157941818237 time to delete rle : 0.0012469291687011719 batch 1 Loaded 15 chid ids of type : 3726 Number RLEs to save : 11599 TO DO : save crop sub photo not yet done ! save time : 0.6666140556335449 map_output_result : {1336616382: (0.0, 'Should be the crop_list due to order', 0.0), 1336616380: (0.0, 'Should be the crop_list due to order', 0.0), 1336616377: (0.0, 'Should be the crop_list due to order', 0.0), 1336616376: (0.0, 'Should be the crop_list due to order', 0.0), 1336616375: (0.0, 'Should be the crop_list due to order', 0.0), 1336616121: (0.0, 'Should be the crop_list due to order', 0.0), 1336562176: (0.0, 'Should be the crop_list due to order', 0.0), 1336562174: (0.0, 'Should be the crop_list due to order', 0.0), 1336562169: (0.0, 'Should be the crop_list due to order', 0.0), 1336554334: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1336616382, 1336616380, 1336616377, 1336616376, 1336616375, 1336616121, 1336562176, 1336562174, 1336562169, 1336554334] Looping around the photos to save general results len do output : 10 /1336616382.Didn't retrieve data . /1336616380.Didn't retrieve data . /1336616377.Didn't retrieve data . /1336616376.Didn't retrieve data . /1336616375.Didn't retrieve data . /1336616121.Didn't retrieve data . /1336562176.Didn't retrieve data . /1336562174.Didn't retrieve data . /1336562169.Didn't retrieve data . /1336554334.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616382', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616380', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616377', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616376', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616375', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616121', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336562176', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336562174', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336562169', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336554334', None, None, None, None, None, '2574201') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013716459274291992 save_final save missing photos in datou_result : time spend for datou_step_exec : 244.72231531143188 time spend to save output : 0.014190196990966797 total time spend for step 5 : 244.73650550842285 step6:crop_condition Tue Feb 11 00:10:48 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 171 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739229051_376160 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.368582010269165 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! About to upload 5 photos upload in portfolio : 4869462 init cache_photo without model_param we have 5 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739229057_376160 we have uploaded 5 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.42976713180542 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739229060_376160 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.4785668849945068 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739229064_376160 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.7990925312042236 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! About to upload 5 photos upload in portfolio : 4869462 init cache_photo without model_param we have 5 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739229067_376160 we have uploaded 5 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.3679156303405762 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739229071_376160 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.8224039077758789 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1336616382, 1336616380, 1336616377, 1336616376, 1336616375, 1336616121, 1336562176, 1336562174, 1336562169, 1336554334] Looping around the photos to save general results len do output : 22 /1336626526Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336626527Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336626528Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336626529Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336626533Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336626534Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336626535Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336626536Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336626537Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336626539Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336626541Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336626542Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336626543Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336626544Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336626545Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336626549Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336626550Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336626551Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336626552Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336626553Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336626557Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336626558Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616382', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616380', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616377', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616376', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616375', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616121', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336562176', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336562174', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336562169', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336554334', None, None, None, None, None, '2574201') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 76 time used for this insertion : 0.034339189529418945 save_final save missing photos in datou_result : time spend for datou_step_exec : 24.218997478485107 time spend to save output : 0.03519773483276367 total time spend for step 6 : 24.25419521331787 step7:ventilate_hashtags_in_portfolio Tue Feb 11 00:11:12 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20423281 get user id for portfolio 20423281 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20423281 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('environnement','Teint_Dans_La_Masse','papier','metal','Carton_brun','flou','plastique','autre_refus','kraft','cartonnette','Carton_gris','mal_croppe')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20423281 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('environnement','Teint_Dans_La_Masse','papier','metal','Carton_brun','flou','plastique','autre_refus','kraft','cartonnette','Carton_gris','mal_croppe')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20423281 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('environnement','Teint_Dans_La_Masse','papier','metal','Carton_brun','flou','plastique','autre_refus','kraft','cartonnette','Carton_gris','mal_croppe')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20423441,20423442,20423443,20423444,20423445,20423446,20423447,20423448,20423449,20423450,20423451,20423452?tags=environnement,Teint_Dans_La_Masse,papier,metal,Carton_brun,flou,plastique,autre_refus,kraft,cartonnette,Carton_gris,mal_croppe Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1336616382, 1336616380, 1336616377, 1336616376, 1336616375, 1336616121, 1336562176, 1336562174, 1336562169, 1336554334] Looping around the photos to save general results len do output : 1 /20423281. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616382', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616380', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616377', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616376', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616375', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616121', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336562176', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336562174', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336562169', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336554334', None, None, None, None, None, '2574201') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.013767480850219727 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.6878201961517334 time spend to save output : 0.014100313186645508 total time spend for step 7 : 1.701920509338379 step8:final Tue Feb 11 00:11:14 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1336616382: ('0.08109472723835193',), 1336616380: ('0.08109472723835193',), 1336616377: ('0.08109472723835193',), 1336616376: ('0.08109472723835193',), 1336616375: ('0.08109472723835193',), 1336616121: ('0.08109472723835193',), 1336562176: ('0.08109472723835193',), 1336562174: ('0.08109472723835193',), 1336562169: ('0.08109472723835193',), 1336554334: ('0.08109472723835193',)} new output for save of step final : {1336616382: ('0.08109472723835193',), 1336616380: ('0.08109472723835193',), 1336616377: ('0.08109472723835193',), 1336616376: ('0.08109472723835193',), 1336616375: ('0.08109472723835193',), 1336616121: ('0.08109472723835193',), 1336562176: ('0.08109472723835193',), 1336562174: ('0.08109472723835193',), 1336562169: ('0.08109472723835193',), 1336554334: ('0.08109472723835193',)} [1336616382, 1336616380, 1336616377, 1336616376, 1336616375, 1336616121, 1336562176, 1336562174, 1336562169, 1336554334] Looping around the photos to save general results len do output : 10 /1336616382.Didn't retrieve data . /1336616380.Didn't retrieve data . /1336616377.Didn't retrieve data . /1336616376.Didn't retrieve data . /1336616375.Didn't retrieve data . /1336616121.Didn't retrieve data . /1336562176.Didn't retrieve data . /1336562174.Didn't retrieve data . /1336562169.Didn't retrieve data . /1336554334.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616382', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616380', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616377', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616376', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616375', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616121', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336562176', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336562174', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336562169', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336554334', None, None, None, None, None, '2574201') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013359546661376953 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.10258722305297852 time spend to save output : 0.013884305953979492 total time spend for step 8 : 0.11647152900695801 step9:velours_tree Tue Feb 11 00:11:14 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.09439778327941895 time spend to save output : 5.14984130859375e-05 total time spend for step 9 : 0.09444928169250488 step10:send_mail_cod Tue Feb 11 00:11:14 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20423281_11-02-2025_00_11_14.pdf 20423444 imagette204234441739229074 20423446 imagette204234461739229074 20423447 change filename to text .change filename to text .imagette204234471739229074 20423448 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette204234481739229074 20423449 imagette204234491739229075 20423450 change filename to text .change filename to text .imagette204234501739229075 20423452 imagette204234521739229075 20423442 change filename to text .change filename to text .change filename to text .change filename to text .imagette204234421739229075 20423445 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette204234451739229075 20423451 change filename to text .change filename to text .change filename to text .change filename to text .imagette204234511739229076 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20423281 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20423441,20423442,20423443,20423444,20423445,20423446,20423447,20423448,20423449,20423450,20423451,20423452?tags=environnement,Teint_Dans_La_Masse,papier,metal,Carton_brun,flou,plastique,autre_refus,kraft,cartonnette,Carton_gris,mal_croppe your option no_mail is active, we will not send the real mail to your client args[1336616382] : ('0.08109472723835193',) no score found for photo 1336616382 apple ('0.08109472723835193',) We are sending mail with results at report@fotonower.com args[1336616380] : ('0.08109472723835193',) no score found for photo 1336616380 apple ('0.08109472723835193',) We are sending mail with results at report@fotonower.com args[1336616377] : ('0.08109472723835193',) no score found for photo 1336616377 apple ('0.08109472723835193',) We are sending mail with results at report@fotonower.com args[1336616376] : ('0.08109472723835193',) no score found for photo 1336616376 apple ('0.08109472723835193',) We are sending mail with results at report@fotonower.com args[1336616375] : ('0.08109472723835193',) no score found for photo 1336616375 apple ('0.08109472723835193',) We are sending mail with results at report@fotonower.com args[1336616121] : ('0.08109472723835193',) no score found for photo 1336616121 apple ('0.08109472723835193',) We are sending mail with results at report@fotonower.com args[1336562176] : ('0.08109472723835193',) no score found for photo 1336562176 apple ('0.08109472723835193',) We are sending mail with results at report@fotonower.com args[1336562174] : ('0.08109472723835193',) no score found for photo 1336562174 apple ('0.08109472723835193',) We are sending mail with results at report@fotonower.com args[1336562169] : ('0.08109472723835193',) no score found for photo 1336562169 apple ('0.08109472723835193',) We are sending mail with results at report@fotonower.com args[1336554334] : ('0.08109472723835193',) no score found for photo 1336554334 apple ('0.08109472723835193',) We are sending mail with results at report@fotonower.com refus_total : 0.08109472723835193 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20423281 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1336548735,1336616377,1336616376,1336562176,1336554334,1336554248,1336554246,1336554237,1336554231,1336554227,1336549213,1336549210,1336549207,1336549205,1336549004,1336548902,1336548726,1336548792,1336548746,1336616382) Found this number of photos: 20 begin to download photo : 1336548735 begin to download photo : 1336554248 begin to download photo : 1336549213 begin to download photo : 1336548902 download finish for photo 1336548735 begin to download photo : 1336616377 download finish for photo 1336548902 begin to download photo : 1336548726 download finish for photo 1336554248 begin to download photo : 1336554246 download finish for photo 1336549213 begin to download photo : 1336549210 download finish for photo 1336616377 begin to download photo : 1336616376 download finish for photo 1336554246 begin to download photo : 1336554237 download finish for photo 1336548726 begin to download photo : 1336548792 download finish for photo 1336616376 begin to download photo : 1336562176 download finish for photo 1336549210 begin to download photo : 1336549207 download finish for photo 1336554237 begin to download photo : 1336554231 download finish for photo 1336548792 begin to download photo : 1336548746 download finish for photo 1336562176 begin to download photo : 1336554334 download finish for photo 1336549207 begin to download photo : 1336549205 download finish for photo 1336554231 begin to download photo : 1336554227 download finish for photo 1336554334 download finish for photo 1336548746 begin to download photo : 1336616382 download finish for photo 1336549205 begin to download photo : 1336549004 download finish for photo 1336554227 download finish for photo 1336616382 download finish for photo 1336549004 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20423281_11-02-2025_00_11_14.pdf results_Auto_P20423281_11-02-2025_00_11_14.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20423281_11-02-2025_00_11_14.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20423281','results_Auto_P20423281_11-02-2025_00_11_14.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20423281_11-02-2025_00_11_14.pdf','pdf','','0.76','0.08109472723835193') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1336616382, 1336616380, 1336616377, 1336616376, 1336616375, 1336616121, 1336562176, 1336562174, 1336562169, 1336554334] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616382', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616380', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616377', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616376', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616375', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616121', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336562176', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336562174', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336562169', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336554334', None, None, None, None, None, '2574201') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.013646125793457031 save_final save missing photos in datou_result : time spend for datou_step_exec : 6.185904264450073 time spend to save output : 0.013919353485107422 total time spend for step 10 : 6.199823617935181 step11:split_time_score Tue Feb 11 00:11:20 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('17', 18), ('18', 9)) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 10022025 20423281 Nombre de photos uploadées : 27 / 23040 (0%) 10022025 20423281 Nombre de photos taguées (types de déchets): 0 / 27 (0%) 10022025 20423281 Nombre de photos taguées (volume) : 0 / 27 (0%) elapsed_time : load_data_split_time_score 1.9073486328125e-06 elapsed_time : order_list_meta_photo_and_scores 5.7220458984375e-06 ??????????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.001451253890991211 elapsed_time : insert_dashboard_record_day_entry 0.025477170944213867 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423264 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423265 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423266 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423267 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423268 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423269 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423270 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423271 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423272 order by id desc limit 1 Qualite : 0.03219250657314503 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20406976_10-02-2025_18_25_17.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20406976 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11560 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11567 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11567 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11563 crop_condition is not consistent : 4 used against 2 in the step definition ! Step 11563 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11564 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11564 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11573 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11573 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11566 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11566 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 1 of step 11560 have datatype=2 whereas input 1 of step 11564 have datatype=7 WARNING : type of output 2 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11565 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11567 doesn't seem to be define in the database( WARNING : type of input 3 of step 11563 doesn't seem to be define in the database( WARNING : type of output 3 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11568 doesn't seem to be define in the database( WARNING : type of output 1 of step 11568 doesn't seem to be define in the database( WARNING : type of input 3 of step 11566 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11570 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11569 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11570 doesn't seem to be define in the database( WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11569 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11568 have datatype=10 whereas input 3 of step 11571 have datatype=6 WARNING : type of input 2 of step 11573 doesn't seem to be define in the database( WARNING : output 1 of step 11565 have datatype=7 whereas input 2 of step 11573 have datatype=None WARNING : type of output 3 of step 11573 doesn't seem to be define in the database( WARNING : type of input 3 of step 11568 doesn't seem to be define in the database( WARNING : output 0 of step 11568 have datatype=10 whereas input 0 of step 11587 have datatype=18 WARNING : type of input 5 of step 11571 doesn't seem to be define in the database( WARNING : output 0 of step 11587 have datatype=11 whereas input 5 of step 11571 have datatype=None WARNING : output 0 of step 11564 have datatype=1 whereas input 0 of step 11565 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20406976 AND mptpi.`type`=3327 To do Qualite : 0.06874626735493077 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20406978_10-02-2025_19_56_28.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20406978 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20406978 AND mptpi.`type`=4207 To do Qualite : 0.039250421093410016 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20406979_10-02-2025_17_47_45.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20406979 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20406979 AND mptpi.`type`=4207 To do Qualite : 0.08218852237472649 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20406980_10-02-2025_15_39_20.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20406980 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20406980 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20406981 order by id desc limit 1 Qualite : 0.22224855111315364 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20406982_10-02-2025_16_37_17.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20406982 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20406982 AND mptpi.`type`=3594 To do Qualite : 0.5242561690199244 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20406983_10-02-2025_22_51_23.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20406983 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11548 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11556 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11557 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11552 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11552 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11555 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11555 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11580 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11580 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11551 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11550 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11550 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11559 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11551 doesn't seem to be define in the database( WARNING : type of input 3 of step 11550 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11548 doesn't seem to be define in the database( WARNING : type of input 2 of step 11552 doesn't seem to be define in the database( WARNING : output 1 of step 11548 have datatype=2 whereas input 1 of step 11555 have datatype=7 WARNING : type of output 2 of step 11555 doesn't seem to be define in the database( WARNING : type of input 1 of step 11549 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11551 have datatype=10 whereas input 3 of step 11558 have datatype=6 WARNING : type of input 2 of step 11580 doesn't seem to be define in the database( WARNING : output 1 of step 11549 have datatype=7 whereas input 2 of step 11580 have datatype=None WARNING : type of output 3 of step 11580 doesn't seem to be define in the database( WARNING : type of input 1 of step 11551 doesn't seem to be define in the database( WARNING : output 0 of step 11551 have datatype=10 whereas input 0 of step 11586 have datatype=18 WARNING : type of input 5 of step 11558 doesn't seem to be define in the database( WARNING : output 0 of step 11586 have datatype=11 whereas input 5 of step 11558 have datatype=None WARNING : type of output 1 of step 11556 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : type of output 1 of step 11557 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : output 0 of step 11555 have datatype=1 whereas input 0 of step 11549 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20406983 AND mptpi.`type`=4200 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20406984 order by id desc limit 1 Qualite : 0.08183746641303798 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20406985_10-02-2025_17_32_10.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20406985 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20406985 AND mptpi.`type`=4207 To do Qualite : 0.059440007083548574 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20406986_10-02-2025_16_45_07.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20406986 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20406986 AND mptpi.`type`=4207 To do Qualite : 0.062017376388549365 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20406987_10-02-2025_15_54_08.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20406987 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20406987 AND mptpi.`type`=4207 To do Qualite : 0.06428706812306653 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20406988_10-02-2025_15_35_18.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20406988 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20406988 AND mptpi.`type`=4207 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423273 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423274 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423275 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423276 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423277 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423278 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423279 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423280 order by id desc limit 1 Qualite : 0.08109472723835193 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20423281_11-02-2025_00_11_14.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423281 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20423281 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423282 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423283 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423284 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423285 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423286 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423287 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423288 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423289 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423290 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423291 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423292 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20423293 order by id desc limit 1 NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'10022025': {'nb_upload': 27, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1336616382, 1336616380, 1336616377, 1336616376, 1336616375, 1336616121, 1336562176, 1336562174, 1336562169, 1336554334] Looping around the photos to save general results len do output : 1 /20423281Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616382', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616380', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616377', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616376', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616375', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336616121', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336562176', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336562174', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336562169', None, None, None, None, None, '2574201') ('3459', None, None, None, None, None, None, None, '2574201') ('3459', '20423281', '1336554334', None, None, None, None, None, '2574201') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.01416468620300293 save_final save missing photos in datou_result : time spend for datou_step_exec : 25.585038661956787 time spend to save output : 0.014451265335083008 total time spend for step 11 : 25.59948992729187 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 157.07user 243.35system 7:21.56elapsed 90%CPU (0avgtext+0avgdata 4207164maxresident)k 2678104inputs+132288outputs (9130major+14276288minor)pagefaults 0swaps