python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3701782 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 28>10 please execute split_portfolio.py -i 20254669 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2551093'] with mtr_portfolio_ids : ['20254669'] and first list_photo_ids : [] new path : /proc/3701782/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 2.245917558670044 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Wed Feb 5 12:56:31 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5485 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-05 12:56:36.384263: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-05 12:56:36.419064: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-05 12:56:36.421221: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f5188000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-05 12:56:36.421272: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-05 12:56:36.425240: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-05 12:56:36.555102: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x33155130 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-05 12:56:36.555143: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-05 12:56:36.556286: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-05 12:56:36.556617: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-05 12:56:36.559862: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-05 12:56:36.575483: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-05 12:56:36.576688: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-05 12:56:36.582470: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-05 12:56:36.585230: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-05 12:56:36.603241: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-05 12:56:36.604426: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-05 12:56:36.604866: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-05 12:56:36.606040: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-05 12:56:36.606057: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-05 12:56:36.606066: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-05 12:56:36.607740: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4849 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-05 12:56:37.005890: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-05 12:56:37.005997: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-05 12:56:37.006020: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-05 12:56:37.006039: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-05 12:56:37.006065: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-05 12:56:37.006083: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-05 12:56:37.006100: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-05 12:56:37.006118: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-05 12:56:37.007298: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-05 12:56:37.008470: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-05 12:56:37.008518: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-05 12:56:37.008536: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-05 12:56:37.008553: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-05 12:56:37.008570: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-05 12:56:37.008587: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-05 12:56:37.008603: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-05 12:56:37.008620: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-05 12:56:37.009633: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-05 12:56:37.009663: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-05 12:56:37.009672: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-05 12:56:37.009679: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-05 12:56:37.010687: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4849 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-05 12:56:42.386982: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 51380224 exceeds 10% of free system memory. 2025-02-05 12:56:49.381608: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-05 12:56:49.697442: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-05 12:56:53.029804: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 32 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 16 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 16 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 26 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 25 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 24 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 20 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 17 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 17 Detection mask done ! Trying to reset tf kernel 3702905 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 383 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5264 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.005990028381347656 nb_pixel_total : 143588 time to create 1 rle with old method : 0.15965056419372559 length of segment : 591 time for calcul the mask position with numpy : 0.0011491775512695312 nb_pixel_total : 30332 time to create 1 rle with old method : 0.03583884239196777 length of segment : 239 time for calcul the mask position with numpy : 0.0009772777557373047 nb_pixel_total : 49515 time to create 1 rle with old method : 0.06154608726501465 length of segment : 312 time for calcul the mask position with numpy : 0.0004699230194091797 nb_pixel_total : 7029 time to create 1 rle with old method : 0.00980997085571289 length of segment : 127 time for calcul the mask position with numpy : 0.002343416213989258 nb_pixel_total : 87004 time to create 1 rle with old method : 0.11114358901977539 length of segment : 477 time for calcul the mask position with numpy : 0.0004477500915527344 nb_pixel_total : 3710 time to create 1 rle with old method : 0.005134105682373047 length of segment : 115 time for calcul the mask position with numpy : 0.00025391578674316406 nb_pixel_total : 4521 time to create 1 rle with old method : 0.006394386291503906 length of segment : 77 time for calcul the mask position with numpy : 0.0004246234893798828 nb_pixel_total : 14937 time to create 1 rle with old method : 0.01872992515563965 length of segment : 147 time for calcul the mask position with numpy : 0.0003135204315185547 nb_pixel_total : 7088 time to create 1 rle with old method : 0.010035037994384766 length of segment : 138 time for calcul the mask position with numpy : 0.00014472007751464844 nb_pixel_total : 1261 time to create 1 rle with old method : 0.0018429756164550781 length of segment : 45 time for calcul the mask position with numpy : 0.0007011890411376953 nb_pixel_total : 18608 time to create 1 rle with old method : 0.024565696716308594 length of segment : 198 time for calcul the mask position with numpy : 0.00014781951904296875 nb_pixel_total : 2242 time to create 1 rle with old method : 0.003175973892211914 length of segment : 49 time for calcul the mask position with numpy : 0.00010085105895996094 nb_pixel_total : 931 time to create 1 rle with old method : 0.0014503002166748047 length of segment : 37 time for calcul the mask position with numpy : 0.0014901161193847656 nb_pixel_total : 61750 time to create 1 rle with old method : 0.0782022476196289 length of segment : 383 time for calcul the mask position with numpy : 0.0004305839538574219 nb_pixel_total : 7491 time to create 1 rle with old method : 0.010083436965942383 length of segment : 157 time for calcul the mask position with numpy : 0.001512289047241211 nb_pixel_total : 53890 time to create 1 rle with old method : 0.07033109664916992 length of segment : 210 time for calcul the mask position with numpy : 0.007235288619995117 nb_pixel_total : 210658 time to create 1 rle with new method : 0.013880729675292969 length of segment : 658 time for calcul the mask position with numpy : 0.0002944469451904297 nb_pixel_total : 6013 time to create 1 rle with old method : 0.007399797439575195 length of segment : 68 time for calcul the mask position with numpy : 0.0003955364227294922 nb_pixel_total : 12904 time to create 1 rle with old method : 0.01756119728088379 length of segment : 220 time for calcul the mask position with numpy : 0.0008723735809326172 nb_pixel_total : 28989 time to create 1 rle with old method : 0.03762698173522949 length of segment : 317 time for calcul the mask position with numpy : 0.0014982223510742188 nb_pixel_total : 47406 time to create 1 rle with old method : 0.06369161605834961 length of segment : 315 time for calcul the mask position with numpy : 0.00011444091796875 nb_pixel_total : 1723 time to create 1 rle with old method : 0.002508401870727539 length of segment : 73 time for calcul the mask position with numpy : 0.00019216537475585938 nb_pixel_total : 5844 time to create 1 rle with old method : 0.00797271728515625 length of segment : 83 time for calcul the mask position with numpy : 0.0003826618194580078 nb_pixel_total : 7624 time to create 1 rle with old method : 0.009966611862182617 length of segment : 99 time for calcul the mask position with numpy : 0.0040667057037353516 nb_pixel_total : 228977 time to create 1 rle with new method : 0.01676034927368164 length of segment : 666 time for calcul the mask position with numpy : 0.0004429817199707031 nb_pixel_total : 13889 time to create 1 rle with old method : 0.017438411712646484 length of segment : 206 time for calcul the mask position with numpy : 0.0003502368927001953 nb_pixel_total : 10296 time to create 1 rle with old method : 0.013716936111450195 length of segment : 101 time for calcul the mask position with numpy : 0.0010988712310791016 nb_pixel_total : 39814 time to create 1 rle with old method : 0.05083799362182617 length of segment : 268 time for calcul the mask position with numpy : 0.005445003509521484 nb_pixel_total : 148001 time to create 1 rle with old method : 0.16953134536743164 length of segment : 735 time for calcul the mask position with numpy : 0.004583835601806641 nb_pixel_total : 69404 time to create 1 rle with old method : 0.08111691474914551 length of segment : 599 time for calcul the mask position with numpy : 0.001132965087890625 nb_pixel_total : 54532 time to create 1 rle with old method : 0.061933040618896484 length of segment : 606 time for calcul the mask position with numpy : 0.0007672309875488281 nb_pixel_total : 49492 time to create 1 rle with old method : 0.06665492057800293 length of segment : 261 time for calcul the mask position with numpy : 0.000812530517578125 nb_pixel_total : 50836 time to create 1 rle with old method : 0.07148575782775879 length of segment : 326 time for calcul the mask position with numpy : 0.0005934238433837891 nb_pixel_total : 36733 time to create 1 rle with old method : 0.046666860580444336 length of segment : 230 time for calcul the mask position with numpy : 0.00019407272338867188 nb_pixel_total : 5479 time to create 1 rle with old method : 0.007666826248168945 length of segment : 143 time for calcul the mask position with numpy : 0.001338958740234375 nb_pixel_total : 83110 time to create 1 rle with old method : 0.1085810661315918 length of segment : 376 time for calcul the mask position with numpy : 0.0002741813659667969 nb_pixel_total : 13533 time to create 1 rle with old method : 0.0182187557220459 length of segment : 192 time for calcul the mask position with numpy : 0.00029754638671875 nb_pixel_total : 11802 time to create 1 rle with old method : 0.014127016067504883 length of segment : 114 time for calcul the mask position with numpy : 0.00020599365234375 nb_pixel_total : 6360 time to create 1 rle with old method : 0.009226560592651367 length of segment : 78 time for calcul the mask position with numpy : 0.00021910667419433594 nb_pixel_total : 11268 time to create 1 rle with old method : 0.014933586120605469 length of segment : 138 time for calcul the mask position with numpy : 0.0003981590270996094 nb_pixel_total : 14096 time to create 1 rle with old method : 0.018764019012451172 length of segment : 190 time for calcul the mask position with numpy : 0.00021767616271972656 nb_pixel_total : 12123 time to create 1 rle with old method : 0.016586780548095703 length of segment : 96 time for calcul the mask position with numpy : 0.0002453327178955078 nb_pixel_total : 12261 time to create 1 rle with old method : 0.0165560245513916 length of segment : 144 time for calcul the mask position with numpy : 0.000652313232421875 nb_pixel_total : 28661 time to create 1 rle with old method : 0.03659391403198242 length of segment : 276 time for calcul the mask position with numpy : 0.0015034675598144531 nb_pixel_total : 97451 time to create 1 rle with old method : 0.11432981491088867 length of segment : 507 time for calcul the mask position with numpy : 0.0005824565887451172 nb_pixel_total : 23741 time to create 1 rle with old method : 0.02826237678527832 length of segment : 217 time for calcul the mask position with numpy : 0.0003724098205566406 nb_pixel_total : 14883 time to create 1 rle with old method : 0.020769596099853516 length of segment : 154 time for calcul the mask position with numpy : 0.00021886825561523438 nb_pixel_total : 7326 time to create 1 rle with old method : 0.010704278945922852 length of segment : 58 time for calcul the mask position with numpy : 0.0008418560028076172 nb_pixel_total : 47866 time to create 1 rle with old method : 0.06421351432800293 length of segment : 316 time for calcul the mask position with numpy : 0.0003478527069091797 nb_pixel_total : 12810 time to create 1 rle with old method : 0.020789623260498047 length of segment : 190 time for calcul the mask position with numpy : 0.001203298568725586 nb_pixel_total : 45869 time to create 1 rle with old method : 0.06261372566223145 length of segment : 287 time for calcul the mask position with numpy : 0.0002925395965576172 nb_pixel_total : 13753 time to create 1 rle with old method : 0.016284704208374023 length of segment : 153 time for calcul the mask position with numpy : 0.0008089542388916016 nb_pixel_total : 45052 time to create 1 rle with old method : 0.059741973876953125 length of segment : 275 time for calcul the mask position with numpy : 0.0008552074432373047 nb_pixel_total : 6300 time to create 1 rle with old method : 0.009019851684570312 length of segment : 204 time for calcul the mask position with numpy : 0.002110004425048828 nb_pixel_total : 132555 time to create 1 rle with old method : 0.15738415718078613 length of segment : 470 time for calcul the mask position with numpy : 0.002528667449951172 nb_pixel_total : 86499 time to create 1 rle with old method : 0.1375751495361328 length of segment : 275 time for calcul the mask position with numpy : 0.0011515617370605469 nb_pixel_total : 30525 time to create 1 rle with old method : 0.034627437591552734 length of segment : 184 time for calcul the mask position with numpy : 0.0007455348968505859 nb_pixel_total : 12890 time to create 1 rle with old method : 0.018077373504638672 length of segment : 195 time for calcul the mask position with numpy : 0.0005114078521728516 nb_pixel_total : 7882 time to create 1 rle with old method : 0.010885953903198242 length of segment : 141 time for calcul the mask position with numpy : 0.001592397689819336 nb_pixel_total : 39741 time to create 1 rle with old method : 0.05086064338684082 length of segment : 259 time for calcul the mask position with numpy : 0.00016188621520996094 nb_pixel_total : 1434 time to create 1 rle with old method : 0.0020499229431152344 length of segment : 65 time for calcul the mask position with numpy : 0.006672859191894531 nb_pixel_total : 189699 time to create 1 rle with new method : 0.01446080207824707 length of segment : 562 time for calcul the mask position with numpy : 0.0008244514465332031 nb_pixel_total : 19641 time to create 1 rle with old method : 0.028970718383789062 length of segment : 101 time for calcul the mask position with numpy : 0.003414154052734375 nb_pixel_total : 112981 time to create 1 rle with old method : 0.14194154739379883 length of segment : 403 time for calcul the mask position with numpy : 0.00800943374633789 nb_pixel_total : 305445 time to create 1 rle with new method : 0.021484375 length of segment : 1243 time for calcul the mask position with numpy : 0.0015211105346679688 nb_pixel_total : 48042 time to create 1 rle with old method : 0.05437636375427246 length of segment : 255 time for calcul the mask position with numpy : 0.00014209747314453125 nb_pixel_total : 782 time to create 1 rle with old method : 0.001088857650756836 length of segment : 33 time for calcul the mask position with numpy : 9.131431579589844e-05 nb_pixel_total : 1657 time to create 1 rle with old method : 0.0025396347045898438 length of segment : 59 time for calcul the mask position with numpy : 0.0021257400512695312 nb_pixel_total : 72475 time to create 1 rle with old method : 0.08112525939941406 length of segment : 444 time for calcul the mask position with numpy : 0.00654149055480957 nb_pixel_total : 206735 time to create 1 rle with new method : 0.013728141784667969 length of segment : 568 time for calcul the mask position with numpy : 0.0071713924407958984 nb_pixel_total : 154692 time to create 1 rle with new method : 0.013248443603515625 length of segment : 485 time for calcul the mask position with numpy : 0.0008096694946289062 nb_pixel_total : 15000 time to create 1 rle with old method : 0.0200045108795166 length of segment : 190 time for calcul the mask position with numpy : 0.0003902912139892578 nb_pixel_total : 3996 time to create 1 rle with old method : 0.005385875701904297 length of segment : 133 time for calcul the mask position with numpy : 0.0021021366119384766 nb_pixel_total : 77416 time to create 1 rle with old method : 0.08545231819152832 length of segment : 424 time for calcul the mask position with numpy : 0.0026128292083740234 nb_pixel_total : 55293 time to create 1 rle with old method : 0.06190776824951172 length of segment : 434 time for calcul the mask position with numpy : 0.003091096878051758 nb_pixel_total : 109821 time to create 1 rle with old method : 0.12037968635559082 length of segment : 451 time for calcul the mask position with numpy : 0.0036971569061279297 nb_pixel_total : 110733 time to create 1 rle with old method : 0.12102174758911133 length of segment : 445 time for calcul the mask position with numpy : 0.0001633167266845703 nb_pixel_total : 991 time to create 1 rle with old method : 0.0014986991882324219 length of segment : 29 time for calcul the mask position with numpy : 0.003040313720703125 nb_pixel_total : 167695 time to create 1 rle with new method : 0.012080669403076172 length of segment : 437 time for calcul the mask position with numpy : 0.0006055831909179688 nb_pixel_total : 11000 time to create 1 rle with old method : 0.014024972915649414 length of segment : 190 time for calcul the mask position with numpy : 0.003415346145629883 nb_pixel_total : 114299 time to create 1 rle with old method : 0.130706787109375 length of segment : 422 time for calcul the mask position with numpy : 0.004900217056274414 nb_pixel_total : 132736 time to create 1 rle with old method : 0.16838860511779785 length of segment : 544 time for calcul the mask position with numpy : 0.0049610137939453125 nb_pixel_total : 195095 time to create 1 rle with new method : 0.011096715927124023 length of segment : 557 time for calcul the mask position with numpy : 0.0036907196044921875 nb_pixel_total : 114736 time to create 1 rle with old method : 0.12491273880004883 length of segment : 457 time for calcul the mask position with numpy : 0.00020956993103027344 nb_pixel_total : 6987 time to create 1 rle with old method : 0.007871627807617188 length of segment : 99 time for calcul the mask position with numpy : 0.0011980533599853516 nb_pixel_total : 38858 time to create 1 rle with old method : 0.04989123344421387 length of segment : 290 time for calcul the mask position with numpy : 0.0004661083221435547 nb_pixel_total : 10792 time to create 1 rle with old method : 0.014450788497924805 length of segment : 91 time for calcul the mask position with numpy : 0.00011968612670898438 nb_pixel_total : 1462 time to create 1 rle with old method : 0.0020897388458251953 length of segment : 61 time for calcul the mask position with numpy : 0.0012214183807373047 nb_pixel_total : 30585 time to create 1 rle with old method : 0.0394895076751709 length of segment : 283 time for calcul the mask position with numpy : 0.0003132820129394531 nb_pixel_total : 4144 time to create 1 rle with old method : 0.005743980407714844 length of segment : 147 time for calcul the mask position with numpy : 0.0009415149688720703 nb_pixel_total : 22191 time to create 1 rle with old method : 0.02895069122314453 length of segment : 328 time for calcul the mask position with numpy : 0.005280017852783203 nb_pixel_total : 186418 time to create 1 rle with new method : 0.01219320297241211 length of segment : 533 time for calcul the mask position with numpy : 0.0023865699768066406 nb_pixel_total : 56437 time to create 1 rle with old method : 0.08991265296936035 length of segment : 448 time for calcul the mask position with numpy : 0.0021889209747314453 nb_pixel_total : 83635 time to create 1 rle with old method : 0.09662151336669922 length of segment : 343 time for calcul the mask position with numpy : 0.0016956329345703125 nb_pixel_total : 57367 time to create 1 rle with old method : 0.07023310661315918 length of segment : 420 time for calcul the mask position with numpy : 0.0022819042205810547 nb_pixel_total : 81055 time to create 1 rle with old method : 0.09978604316711426 length of segment : 471 time for calcul the mask position with numpy : 0.0012085437774658203 nb_pixel_total : 65541 time to create 1 rle with old method : 0.08912992477416992 length of segment : 421 time for calcul the mask position with numpy : 0.004709005355834961 nb_pixel_total : 163496 time to create 1 rle with new method : 0.02402019500732422 length of segment : 709 time for calcul the mask position with numpy : 0.0028791427612304688 nb_pixel_total : 81623 time to create 1 rle with old method : 0.10605978965759277 length of segment : 356 time for calcul the mask position with numpy : 0.0020449161529541016 nb_pixel_total : 81963 time to create 1 rle with old method : 0.10579800605773926 length of segment : 407 time for calcul the mask position with numpy : 0.0026242733001708984 nb_pixel_total : 86503 time to create 1 rle with old method : 0.10497689247131348 length of segment : 542 time for calcul the mask position with numpy : 0.002939462661743164 nb_pixel_total : 116006 time to create 1 rle with old method : 0.13073110580444336 length of segment : 551 time for calcul the mask position with numpy : 0.0005981922149658203 nb_pixel_total : 39936 time to create 1 rle with old method : 0.05194354057312012 length of segment : 215 time for calcul the mask position with numpy : 0.0022950172424316406 nb_pixel_total : 65289 time to create 1 rle with old method : 0.0906686782836914 length of segment : 404 time for calcul the mask position with numpy : 0.00098419189453125 nb_pixel_total : 20951 time to create 1 rle with old method : 0.026921749114990234 length of segment : 264 time for calcul the mask position with numpy : 0.0017881393432617188 nb_pixel_total : 56244 time to create 1 rle with old method : 0.07396531105041504 length of segment : 192 time for calcul the mask position with numpy : 0.004332780838012695 nb_pixel_total : 148913 time to create 1 rle with old method : 0.1909022331237793 length of segment : 403 time for calcul the mask position with numpy : 0.0016090869903564453 nb_pixel_total : 27927 time to create 1 rle with old method : 0.0366818904876709 length of segment : 361 time for calcul the mask position with numpy : 0.00015497207641601562 nb_pixel_total : 1456 time to create 1 rle with old method : 0.0020771026611328125 length of segment : 60 time for calcul the mask position with numpy : 0.0005536079406738281 nb_pixel_total : 15143 time to create 1 rle with old method : 0.02045464515686035 length of segment : 136 time for calcul the mask position with numpy : 0.0001876354217529297 nb_pixel_total : 2755 time to create 1 rle with old method : 0.0038869380950927734 length of segment : 83 time for calcul the mask position with numpy : 0.0033218860626220703 nb_pixel_total : 135271 time to create 1 rle with old method : 0.15243864059448242 length of segment : 644 time for calcul the mask position with numpy : 0.003787994384765625 nb_pixel_total : 155013 time to create 1 rle with new method : 0.01270151138305664 length of segment : 915 time for calcul the mask position with numpy : 0.0005629062652587891 nb_pixel_total : 9481 time to create 1 rle with old method : 0.013314962387084961 length of segment : 175 time for calcul the mask position with numpy : 0.0011181831359863281 nb_pixel_total : 38364 time to create 1 rle with old method : 0.04444766044616699 length of segment : 251 time for calcul the mask position with numpy : 0.0032842159271240234 nb_pixel_total : 134798 time to create 1 rle with old method : 0.15619206428527832 length of segment : 818 time for calcul the mask position with numpy : 0.0006992816925048828 nb_pixel_total : 5136 time to create 1 rle with old method : 0.007520437240600586 length of segment : 264 time for calcul the mask position with numpy : 0.00033855438232421875 nb_pixel_total : 16885 time to create 1 rle with old method : 0.024059772491455078 length of segment : 189 time for calcul the mask position with numpy : 0.0034856796264648438 nb_pixel_total : 108238 time to create 1 rle with old method : 0.12434053421020508 length of segment : 381 time for calcul the mask position with numpy : 0.0037980079650878906 nb_pixel_total : 67070 time to create 1 rle with old method : 0.08164191246032715 length of segment : 668 time for calcul the mask position with numpy : 0.0003216266632080078 nb_pixel_total : 6075 time to create 1 rle with old method : 0.008394479751586914 length of segment : 86 time for calcul the mask position with numpy : 0.008095502853393555 nb_pixel_total : 125081 time to create 1 rle with old method : 0.1373906135559082 length of segment : 664 time for calcul the mask position with numpy : 0.002559661865234375 nb_pixel_total : 95655 time to create 1 rle with old method : 0.12504172325134277 length of segment : 513 time for calcul the mask position with numpy : 0.00042891502380371094 nb_pixel_total : 7609 time to create 1 rle with old method : 0.016921281814575195 length of segment : 91 time for calcul the mask position with numpy : 0.01248621940612793 nb_pixel_total : 340137 time to create 1 rle with new method : 0.16116118431091309 length of segment : 559 time for calcul the mask position with numpy : 0.003173828125 nb_pixel_total : 80945 time to create 1 rle with old method : 0.0954599380493164 length of segment : 623 time for calcul the mask position with numpy : 0.002334117889404297 nb_pixel_total : 75250 time to create 1 rle with old method : 0.08753657341003418 length of segment : 318 time for calcul the mask position with numpy : 0.0003635883331298828 nb_pixel_total : 8301 time to create 1 rle with old method : 0.012922048568725586 length of segment : 119 time for calcul the mask position with numpy : 0.006277561187744141 nb_pixel_total : 270534 time to create 1 rle with new method : 0.03605151176452637 length of segment : 1114 time for calcul the mask position with numpy : 0.003491640090942383 nb_pixel_total : 96171 time to create 1 rle with old method : 0.1492023468017578 length of segment : 447 time for calcul the mask position with numpy : 0.10984206199645996 nb_pixel_total : 1271849 time to create 1 rle with new method : 0.18339157104492188 length of segment : 1777 time for calcul the mask position with numpy : 0.0002944469451904297 nb_pixel_total : 4862 time to create 1 rle with old method : 0.006993293762207031 length of segment : 144 time for calcul the mask position with numpy : 0.004347801208496094 nb_pixel_total : 94727 time to create 1 rle with old method : 0.11292004585266113 length of segment : 726 time for calcul the mask position with numpy : 0.00213623046875 nb_pixel_total : 59478 time to create 1 rle with old method : 0.07931733131408691 length of segment : 570 time for calcul the mask position with numpy : 0.001912832260131836 nb_pixel_total : 74969 time to create 1 rle with old method : 0.09672212600708008 length of segment : 307 time for calcul the mask position with numpy : 0.0005464553833007812 nb_pixel_total : 11138 time to create 1 rle with old method : 0.013041257858276367 length of segment : 159 time for calcul the mask position with numpy : 0.002986907958984375 nb_pixel_total : 125090 time to create 1 rle with old method : 0.16419577598571777 length of segment : 416 time for calcul the mask position with numpy : 0.01846480369567871 nb_pixel_total : 438868 time to create 1 rle with new method : 0.042433738708496094 length of segment : 950 time for calcul the mask position with numpy : 0.00043201446533203125 nb_pixel_total : 10303 time to create 1 rle with old method : 0.014543533325195312 length of segment : 119 time for calcul the mask position with numpy : 0.0008845329284667969 nb_pixel_total : 29477 time to create 1 rle with old method : 0.03452610969543457 length of segment : 257 time for calcul the mask position with numpy : 0.00031685829162597656 nb_pixel_total : 5862 time to create 1 rle with old method : 0.007984638214111328 length of segment : 105 time for calcul the mask position with numpy : 0.0002841949462890625 nb_pixel_total : 5503 time to create 1 rle with old method : 0.0077686309814453125 length of segment : 97 time for calcul the mask position with numpy : 0.0014576911926269531 nb_pixel_total : 36675 time to create 1 rle with old method : 0.043355464935302734 length of segment : 619 time for calcul the mask position with numpy : 0.0011038780212402344 nb_pixel_total : 40552 time to create 1 rle with old method : 0.05063962936401367 length of segment : 274 time for calcul the mask position with numpy : 0.005911588668823242 nb_pixel_total : 186111 time to create 1 rle with new method : 0.006856441497802734 length of segment : 596 time for calcul the mask position with numpy : 0.0004711151123046875 nb_pixel_total : 15268 time to create 1 rle with old method : 0.020077943801879883 length of segment : 227 time for calcul the mask position with numpy : 0.024310588836669922 nb_pixel_total : 711054 time to create 1 rle with new method : 0.2163548469543457 length of segment : 2323 time for calcul the mask position with numpy : 0.011595964431762695 nb_pixel_total : 434058 time to create 1 rle with new method : 0.01602649688720703 length of segment : 727 time for calcul the mask position with numpy : 0.014037847518920898 nb_pixel_total : 504865 time to create 1 rle with new method : 0.05958271026611328 length of segment : 939 time for calcul the mask position with numpy : 0.0038721561431884766 nb_pixel_total : 141751 time to create 1 rle with old method : 0.15723228454589844 length of segment : 558 time for calcul the mask position with numpy : 0.0006375312805175781 nb_pixel_total : 16881 time to create 1 rle with old method : 0.02154994010925293 length of segment : 151 time for calcul the mask position with numpy : 0.002193927764892578 nb_pixel_total : 47923 time to create 1 rle with old method : 0.052524566650390625 length of segment : 605 time for calcul the mask position with numpy : 0.001027822494506836 nb_pixel_total : 15321 time to create 1 rle with old method : 0.020888328552246094 length of segment : 278 time for calcul the mask position with numpy : 0.0016968250274658203 nb_pixel_total : 61470 time to create 1 rle with old method : 0.07757329940795898 length of segment : 364 time for calcul the mask position with numpy : 0.00030231475830078125 nb_pixel_total : 7403 time to create 1 rle with old method : 0.009840250015258789 length of segment : 107 time for calcul the mask position with numpy : 0.0005242824554443359 nb_pixel_total : 23445 time to create 1 rle with old method : 0.0305635929107666 length of segment : 136 time for calcul the mask position with numpy : 0.00043964385986328125 nb_pixel_total : 9116 time to create 1 rle with old method : 0.011887311935424805 length of segment : 135 time for calcul the mask position with numpy : 0.001547098159790039 nb_pixel_total : 74287 time to create 1 rle with old method : 0.08238077163696289 length of segment : 314 time for calcul the mask position with numpy : 0.00032806396484375 nb_pixel_total : 10310 time to create 1 rle with old method : 0.01325535774230957 length of segment : 148 time for calcul the mask position with numpy : 0.0018777847290039062 nb_pixel_total : 101191 time to create 1 rle with old method : 0.1223149299621582 length of segment : 478 time for calcul the mask position with numpy : 0.0009610652923583984 nb_pixel_total : 8830 time to create 1 rle with old method : 0.011182785034179688 length of segment : 75 time for calcul the mask position with numpy : 0.00017571449279785156 nb_pixel_total : 2668 time to create 1 rle with old method : 0.003904104232788086 length of segment : 46 time for calcul the mask position with numpy : 0.0021190643310546875 nb_pixel_total : 76720 time to create 1 rle with old method : 0.09678006172180176 length of segment : 499 time for calcul the mask position with numpy : 0.000225067138671875 nb_pixel_total : 4078 time to create 1 rle with old method : 0.005682706832885742 length of segment : 61 time for calcul the mask position with numpy : 0.0017924308776855469 nb_pixel_total : 41124 time to create 1 rle with old method : 0.04732489585876465 length of segment : 318 time for calcul the mask position with numpy : 0.0027565956115722656 nb_pixel_total : 79588 time to create 1 rle with old method : 0.08605003356933594 length of segment : 494 time for calcul the mask position with numpy : 0.0006859302520751953 nb_pixel_total : 19402 time to create 1 rle with old method : 0.025360107421875 length of segment : 166 time for calcul the mask position with numpy : 0.00027108192443847656 nb_pixel_total : 7498 time to create 1 rle with old method : 0.010302305221557617 length of segment : 65 time for calcul the mask position with numpy : 0.0003173351287841797 nb_pixel_total : 18309 time to create 1 rle with old method : 0.023537635803222656 length of segment : 135 time for calcul the mask position with numpy : 0.0012333393096923828 nb_pixel_total : 67805 time to create 1 rle with old method : 0.08556723594665527 length of segment : 432 time for calcul the mask position with numpy : 0.00022864341735839844 nb_pixel_total : 4032 time to create 1 rle with old method : 0.005563020706176758 length of segment : 77 time for calcul the mask position with numpy : 0.0002899169921875 nb_pixel_total : 4936 time to create 1 rle with old method : 0.006974220275878906 length of segment : 79 time for calcul the mask position with numpy : 0.0004181861877441406 nb_pixel_total : 5926 time to create 1 rle with old method : 0.00805807113647461 length of segment : 178 time for calcul the mask position with numpy : 0.0003666877746582031 nb_pixel_total : 10053 time to create 1 rle with old method : 0.013002872467041016 length of segment : 139 time for calcul the mask position with numpy : 0.0006799697875976562 nb_pixel_total : 20439 time to create 1 rle with old method : 0.027312040328979492 length of segment : 141 time for calcul the mask position with numpy : 0.003780841827392578 nb_pixel_total : 138764 time to create 1 rle with old method : 0.165510892868042 length of segment : 457 time spent for convertir_results : 19.931174278259277 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 179 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Number RLEs to save : 59121 save missing photos in datou_result : time spend for datou_step_exec : 81.82800531387329 time spend to save output : 3.7925126552581787 total time spend for step 1 : 85.62051796913147 step2:crop_condition Wed Feb 5 12:57:57 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 179 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 13 About to insert : list_path_to_insert length 13 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 10 About to insert : list_path_to_insert length 10 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 10 About to insert : list_path_to_insert length 10 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 44 About to insert : list_path_to_insert length 44 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 10 About to insert : list_path_to_insert length 10 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1334497458, 1334497454, 1334497450, 1334497447, 1334497444, 1334497441, 1334497438, 1334497434, 1334497411, 1334497400] Looping around the photos to save general results len do output : 90 /-3653455234Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455231Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455232Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455270Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455261Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455273Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455295Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455312Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455342Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455340Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472530Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472558Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472574Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455258Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455301Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455302Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455304Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455306Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455327Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455349Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472534Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472569Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472571Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455256Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455259Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455309Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455294Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455325Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455314Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455336Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455341Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472541Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472539Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455235Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455260Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455244Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455246Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3655935882Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455254Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455266Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455264Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455263Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455275Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455271Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455279Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455283Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455281Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455284Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455286Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455307Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455300Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455319Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455330Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455324Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455323Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455322Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455311Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455337Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455332Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455333Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455331Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455350Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455343Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455347Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472526Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472540Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472533Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472537Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472538Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472531Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472556Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472549Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472562Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472570Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472580Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472567Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472573Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455238Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455282Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455280Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455338Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472548Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472550Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472544Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472555Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472560Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472572Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455247Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653455287Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3653472527Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497458', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497454', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497450', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497447', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497444', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497441', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497438', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497434', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497411', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497400', None, None, None, None, None, '2551093') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 280 time used for this insertion : 0.03537440299987793 save_final save missing photos in datou_result : time spend for datou_step_exec : 38.273383378982544 time spend to save output : 0.03824305534362793 total time spend for step 2 : 38.31162643432617 step3:thcl Wed Feb 5 12:58:35 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.01591944694519043 time to convert the images to numpy array : 0.34387683868408203 time to import caffe and check if the image exist : 0.013808965682983398 time to convert the images to numpy array : 0.3776562213897705 time to import caffe and check if the image exist : 0.011523008346557617 time to convert the images to numpy array : 0.4065742492675781 time to import caffe and check if the image exist : 0.019840717315673828 time to convert the images to numpy array : 0.4099597930908203 time to import caffe and check if the image exist : 0.023555517196655273 time to convert the images to numpy array : 0.41498374938964844 time to import caffe and check if the image exist : 0.015845775604248047 time to convert the images to numpy array : 0.43476247787475586 time to import caffe and check if the image exist : 0.020308732986450195 time to convert the images to numpy array : 0.4345705509185791 time to import caffe and check if the image exist : 0.02117753028869629 time to convert the images to numpy array : 0.45516085624694824 time to import caffe and check if the image exist : 0.017630338668823242 time to convert the images to numpy array : 0.4596250057220459 time to import caffe and check if the image exist : 0.014858007431030273 time to convert the images to numpy array : 0.4710395336151123 total time to convert the images to numpy array : 1.6210196018218994 list photo_ids error: [] list photo_ids correct : [-3653455246, -3655935882, -3653455254, -3653455266, -3653455264, -3653455263, -3653455275, -3653455271, -3653455279, -3653472562, -3653472570, -3653472580, -3653472567, -3653472573, -3653455238, -3653455282, -3653455280, -3653455338, -3653455234, -3653455231, -3653455232, -3653455270, -3653455261, -3653455273, -3653455295, -3653455312, -3653455342, -3653455283, -3653455281, -3653455284, -3653455286, -3653455307, -3653455300, -3653455319, -3653455330, -3653455324, -3653472548, -3653472550, -3653472544, -3653472555, -3653472560, -3653472572, -3653455247, -3653455287, -3653472527, -3653455340, -3653472530, -3653472558, -3653472574, -3653455258, -3653455301, -3653455302, -3653455304, -3653455306, -3653455325, -3653455314, -3653455336, -3653455341, -3653472541, -3653472539, -3653455235, -3653455260, -3653455244, -3653455323, -3653455322, -3653455311, -3653455337, -3653455332, -3653455333, -3653455331, -3653455350, -3653455343, -3653455347, -3653472526, -3653472540, -3653472533, -3653472537, -3653472538, -3653472531, -3653472556, -3653472549, -3653455327, -3653455349, -3653472534, -3653472569, -3653472571, -3653455256, -3653455259, -3653455309, -3653455294] number of photos to traite : 90 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5485 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5266 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.059347152709961 time used to do the prediction : 0.24505949020385742 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5264 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5264 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 0.9962966442108154 time used to do the prediction : 0.15929603576660156 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 8.821487426757812e-06 save missing photos in datou_result : time spend for datou_step_exec : 11.49952244758606 time spend to save output : 0.0007853507995605469 total time spend for step 3 : 11.50030779838562 step4:merge_mask_thcl_custom Wed Feb 5 12:58:47 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 179 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1334497458, 1334497454, 1334497450, 1334497447, 1334497444, 1334497441, 1334497438, 1334497434, 1334497411, 1334497400] Looping around the photos to save general results len do output : 10 /1334497458Didn't retrieve data .Didn't retrieve data . /1334497454Didn't retrieve data .Didn't retrieve data . /1334497450Didn't retrieve data .Didn't retrieve data . /1334497447Didn't retrieve data .Didn't retrieve data . /1334497444Didn't retrieve data .Didn't retrieve data . /1334497441Didn't retrieve data .Didn't retrieve data . /1334497438Didn't retrieve data .Didn't retrieve data . /1334497434Didn't retrieve data .Didn't retrieve data . /1334497411Didn't retrieve data .Didn't retrieve data . /1334497400Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497458', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497454', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497450', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497447', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497444', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497441', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497438', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497434', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497411', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497400', None, None, None, None, None, '2551093') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013893604278564453 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.06224393844604492 time spend to save output : 0.014356851577758789 total time spend for step 4 : 0.07660079002380371 step5:rle_unique_nms_with_priority Wed Feb 5 12:58:47 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 29 nb_hashtags : 6 time to prepare the origin masks : 11.340747594833374 time for calcul the mask position with numpy : 0.07895517349243164 nb_pixel_total : 5949708 time to create 1 rle with new method : 0.7185602188110352 time for calcul the mask position with numpy : 0.03494739532470703 nb_pixel_total : 1259 time to create 1 rle with old method : 0.00157928466796875 time for calcul the mask position with numpy : 0.02654719352722168 nb_pixel_total : 4476 time to create 1 rle with old method : 0.0058574676513671875 time for calcul the mask position with numpy : 0.030050039291381836 nb_pixel_total : 210508 time to create 1 rle with new method : 0.1226663589477539 time for calcul the mask position with numpy : 0.02669382095336914 nb_pixel_total : 3689 time to create 1 rle with old method : 0.004847049713134766 time for calcul the mask position with numpy : 0.028463363647460938 nb_pixel_total : 19665 time to create 1 rle with old method : 0.023964405059814453 time for calcul the mask position with numpy : 0.02633810043334961 nb_pixel_total : 2216 time to create 1 rle with old method : 0.0027348995208740234 time for calcul the mask position with numpy : 0.026318073272705078 nb_pixel_total : 920 time to create 1 rle with old method : 0.0011487007141113281 time for calcul the mask position with numpy : 0.02628612518310547 nb_pixel_total : 10244 time to create 1 rle with old method : 0.012194156646728516 time for calcul the mask position with numpy : 0.026454687118530273 nb_pixel_total : 47228 time to create 1 rle with old method : 0.05500674247741699 time for calcul the mask position with numpy : 0.02751469612121582 nb_pixel_total : 922 time to create 1 rle with old method : 0.0012142658233642578 time for calcul the mask position with numpy : 0.028131961822509766 nb_pixel_total : 147575 time to create 1 rle with old method : 0.15331816673278809 time for calcul the mask position with numpy : 0.028122901916503906 nb_pixel_total : 86846 time to create 1 rle with old method : 0.0922694206237793 time for calcul the mask position with numpy : 0.02716994285583496 nb_pixel_total : 12853 time to create 1 rle with old method : 0.015633583068847656 time for calcul the mask position with numpy : 0.02720928192138672 nb_pixel_total : 18773 time to create 1 rle with old method : 0.0228574275970459 time for calcul the mask position with numpy : 0.027110815048217773 nb_pixel_total : 14902 time to create 1 rle with old method : 0.018668413162231445 time for calcul the mask position with numpy : 0.027857303619384766 nb_pixel_total : 30051 time to create 1 rle with old method : 0.03624892234802246 time for calcul the mask position with numpy : 0.02767324447631836 nb_pixel_total : 13846 time to create 1 rle with old method : 0.01698613166809082 time for calcul the mask position with numpy : 0.027567386627197266 nb_pixel_total : 74968 time to create 1 rle with old method : 0.08996105194091797 time for calcul the mask position with numpy : 0.02920985221862793 nb_pixel_total : 53549 time to create 1 rle with old method : 0.0686182975769043 time for calcul the mask position with numpy : 0.02917170524597168 nb_pixel_total : 5969 time to create 1 rle with old method : 0.007959604263305664 time for calcul the mask position with numpy : 0.028265714645385742 nb_pixel_total : 7474 time to create 1 rle with old method : 0.009330272674560547 time for calcul the mask position with numpy : 0.02899479866027832 nb_pixel_total : 143532 time to create 1 rle with old method : 0.15938758850097656 time for calcul the mask position with numpy : 0.028803586959838867 nb_pixel_total : 7009 time to create 1 rle with old method : 0.008496522903442383 time for calcul the mask position with numpy : 0.02868342399597168 nb_pixel_total : 18551 time to create 1 rle with old method : 0.02071976661682129 time for calcul the mask position with numpy : 0.02858257293701172 nb_pixel_total : 61683 time to create 1 rle with old method : 0.06618380546569824 time for calcul the mask position with numpy : 0.02821826934814453 nb_pixel_total : 39660 time to create 1 rle with old method : 0.05930685997009277 time for calcul the mask position with numpy : 0.03274059295654297 nb_pixel_total : 49273 time to create 1 rle with old method : 0.07430338859558105 time for calcul the mask position with numpy : 0.02739262580871582 nb_pixel_total : 5809 time to create 1 rle with old method : 0.007398128509521484 time for calcul the mask position with numpy : 0.02762293815612793 nb_pixel_total : 7082 time to create 1 rle with old method : 0.009076833724975586 create new chi : 2.817410707473755 time to delete rle : 0.015827178955078125 batch 1 Loaded 31 chid ids of type : 3726 Number RLEs to save : 16684 TO DO : save crop sub photo not yet done ! save time : 0.9904911518096924 nb_obj : 14 nb_hashtags : 5 time to prepare the origin masks : 8.772877216339111 time for calcul the mask position with numpy : 0.1925363540649414 nb_pixel_total : 6595190 time to create 1 rle with new method : 0.6146917343139648 time for calcul the mask position with numpy : 0.02095961570739746 nb_pixel_total : 51554 time to create 1 rle with old method : 0.05734109878540039 time for calcul the mask position with numpy : 0.021297454833984375 nb_pixel_total : 83023 time to create 1 rle with old method : 0.10348200798034668 time for calcul the mask position with numpy : 0.020281553268432617 nb_pixel_total : 11207 time to create 1 rle with old method : 0.012333869934082031 time for calcul the mask position with numpy : 0.020649433135986328 nb_pixel_total : 12188 time to create 1 rle with old method : 0.0157315731048584 time for calcul the mask position with numpy : 0.02161407470703125 nb_pixel_total : 12000 time to create 1 rle with old method : 0.015878915786743164 time for calcul the mask position with numpy : 0.022150516510009766 nb_pixel_total : 6284 time to create 1 rle with old method : 0.00823974609375 time for calcul the mask position with numpy : 0.022324562072753906 nb_pixel_total : 14056 time to create 1 rle with old method : 0.01818227767944336 time for calcul the mask position with numpy : 0.021874189376831055 nb_pixel_total : 11673 time to create 1 rle with old method : 0.014748573303222656 time for calcul the mask position with numpy : 0.025002241134643555 nb_pixel_total : 97409 time to create 1 rle with old method : 0.12138938903808594 time for calcul the mask position with numpy : 0.031047582626342773 nb_pixel_total : 36663 time to create 1 rle with old method : 0.046204566955566406 time for calcul the mask position with numpy : 0.03428459167480469 nb_pixel_total : 13491 time to create 1 rle with old method : 0.01765584945678711 time for calcul the mask position with numpy : 0.034169673919677734 nb_pixel_total : 5456 time to create 1 rle with old method : 0.0063517093658447266 time for calcul the mask position with numpy : 0.033184051513671875 nb_pixel_total : 50768 time to create 1 rle with old method : 0.06349301338195801 time for calcul the mask position with numpy : 0.036328792572021484 nb_pixel_total : 49278 time to create 1 rle with old method : 0.09223055839538574 create new chi : 1.8011891841888428 time to delete rle : 0.0009272098541259766 batch 1 Loaded 15 chid ids of type : 3726 Number RLEs to save : 8699 TO DO : save crop sub photo not yet done ! save time : 0.5652339458465576 nb_obj : 11 nb_hashtags : 5 time to prepare the origin masks : 5.401748180389404 time for calcul the mask position with numpy : 0.3798973560333252 nb_pixel_total : 6719759 time to create 1 rle with new method : 0.7497217655181885 time for calcul the mask position with numpy : 0.019339561462402344 nb_pixel_total : 45581 time to create 1 rle with old method : 0.04899740219116211 time for calcul the mask position with numpy : 0.019349098205566406 nb_pixel_total : 12772 time to create 1 rle with old method : 0.016422748565673828 time for calcul the mask position with numpy : 0.020984888076782227 nb_pixel_total : 86148 time to create 1 rle with old method : 0.09135675430297852 time for calcul the mask position with numpy : 0.019633769989013672 nb_pixel_total : 11339 time to create 1 rle with old method : 0.01390385627746582 time for calcul the mask position with numpy : 0.02054142951965332 nb_pixel_total : 47763 time to create 1 rle with old method : 0.057778120040893555 time for calcul the mask position with numpy : 0.021146774291992188 nb_pixel_total : 30438 time to create 1 rle with old method : 0.036401987075805664 time for calcul the mask position with numpy : 0.020694255828857422 nb_pixel_total : 5962 time to create 1 rle with old method : 0.007407188415527344 time for calcul the mask position with numpy : 0.02099013328552246 nb_pixel_total : 44951 time to create 1 rle with old method : 0.05088949203491211 time for calcul the mask position with numpy : 0.021780967712402344 nb_pixel_total : 7254 time to create 1 rle with old method : 0.009350776672363281 time for calcul the mask position with numpy : 0.022745370864868164 nb_pixel_total : 14776 time to create 1 rle with old method : 0.01934528350830078 time for calcul the mask position with numpy : 0.02176666259765625 nb_pixel_total : 23497 time to create 1 rle with old method : 0.025925397872924805 create new chi : 1.7707207202911377 time to delete rle : 0.0007812976837158203 batch 1 Loaded 12 chid ids of type : 3726 Number RLEs to save : 6686 TO DO : save crop sub photo not yet done ! save time : 0.4790060520172119 nb_obj : 21 nb_hashtags : 4 time to prepare the origin masks : 8.054107904434204 time for calcul the mask position with numpy : 0.2110435962677002 nb_pixel_total : 5879908 time to create 1 rle with new method : 0.4947779178619385 time for calcul the mask position with numpy : 0.020406246185302734 nb_pixel_total : 2070 time to create 1 rle with old method : 0.002948760986328125 time for calcul the mask position with numpy : 0.019716262817382812 nb_pixel_total : 987 time to create 1 rle with old method : 0.0012750625610351562 time for calcul the mask position with numpy : 0.0196382999420166 nb_pixel_total : 6607 time to create 1 rle with old method : 0.009191513061523438 time for calcul the mask position with numpy : 0.020395517349243164 nb_pixel_total : 3995 time to create 1 rle with old method : 0.005196332931518555 time for calcul the mask position with numpy : 0.020896434783935547 nb_pixel_total : 112638 time to create 1 rle with old method : 0.12011933326721191 time for calcul the mask position with numpy : 0.02118515968322754 nb_pixel_total : 140117 time to create 1 rle with old method : 0.1594705581665039 time for calcul the mask position with numpy : 0.02133917808532715 nb_pixel_total : 73396 time to create 1 rle with old method : 0.09158444404602051 time for calcul the mask position with numpy : 0.019898414611816406 nb_pixel_total : 1645 time to create 1 rle with old method : 0.002157449722290039 time for calcul the mask position with numpy : 0.020833969116210938 nb_pixel_total : 206271 time to create 1 rle with new method : 0.20995569229125977 time for calcul the mask position with numpy : 0.020234107971191406 nb_pixel_total : 772 time to create 1 rle with old method : 0.001074075698852539 time for calcul the mask position with numpy : 0.02047276496887207 nb_pixel_total : 47959 time to create 1 rle with old method : 0.05975222587585449 time for calcul the mask position with numpy : 0.02089858055114746 nb_pixel_total : 14581 time to create 1 rle with old method : 0.019051313400268555 time for calcul the mask position with numpy : 0.02152252197265625 nb_pixel_total : 19474 time to create 1 rle with old method : 0.0238955020904541 time for calcul the mask position with numpy : 0.020893573760986328 nb_pixel_total : 189617 time to create 1 rle with new method : 0.24568915367126465 time for calcul the mask position with numpy : 0.020938634872436523 nb_pixel_total : 1425 time to create 1 rle with old method : 0.0018656253814697266 time for calcul the mask position with numpy : 0.01973700523376465 nb_pixel_total : 39609 time to create 1 rle with old method : 0.0425417423248291 time for calcul the mask position with numpy : 0.020191669464111328 nb_pixel_total : 110701 time to create 1 rle with old method : 0.13174653053283691 time for calcul the mask position with numpy : 0.01981043815612793 nb_pixel_total : 7821 time to create 1 rle with old method : 0.009739875793457031 time for calcul the mask position with numpy : 0.01975250244140625 nb_pixel_total : 12849 time to create 1 rle with old method : 0.015927791595458984 time for calcul the mask position with numpy : 0.01996779441833496 nb_pixel_total : 23388 time to create 1 rle with old method : 0.029030323028564453 time for calcul the mask position with numpy : 0.020954608917236328 nb_pixel_total : 154410 time to create 1 rle with new method : 0.3461418151855469 create new chi : 2.7419936656951904 time to delete rle : 0.0013470649719238281 batch 1 Loaded 22 chid ids of type : 3726 Number RLEs to save : 13611 TO DO : save crop sub photo not yet done ! save time : 0.7321488857269287 nb_obj : 19 nb_hashtags : 6 time to prepare the origin masks : 7.0170204639434814 time for calcul the mask position with numpy : 0.13286995887756348 nb_pixel_total : 5874222 time to create 1 rle with new method : 0.5766239166259766 time for calcul the mask position with numpy : 0.02076435089111328 nb_pixel_total : 113645 time to create 1 rle with old method : 0.12245583534240723 time for calcul the mask position with numpy : 0.01950836181640625 nb_pixel_total : 14891 time to create 1 rle with old method : 0.018597126007080078 time for calcul the mask position with numpy : 0.01967930793762207 nb_pixel_total : 68779 time to create 1 rle with old method : 0.06910395622253418 time for calcul the mask position with numpy : 0.019866943359375 nb_pixel_total : 114178 time to create 1 rle with old method : 0.11730384826660156 time for calcul the mask position with numpy : 0.01896047592163086 nb_pixel_total : 5553 time to create 1 rle with old method : 0.007161855697631836 time for calcul the mask position with numpy : 0.018956661224365234 nb_pixel_total : 30521 time to create 1 rle with old method : 0.03715181350708008 time for calcul the mask position with numpy : 0.019592761993408203 nb_pixel_total : 81810 time to create 1 rle with old method : 0.08995199203491211 time for calcul the mask position with numpy : 0.019616127014160156 nb_pixel_total : 80920 time to create 1 rle with old method : 0.10288047790527344 time for calcul the mask position with numpy : 0.02146625518798828 nb_pixel_total : 57180 time to create 1 rle with old method : 0.061419010162353516 time for calcul the mask position with numpy : 0.021458148956298828 nb_pixel_total : 22173 time to create 1 rle with old method : 0.02884697914123535 time for calcul the mask position with numpy : 0.02137279510498047 nb_pixel_total : 4126 time to create 1 rle with old method : 0.0052182674407958984 time for calcul the mask position with numpy : 0.02133798599243164 nb_pixel_total : 1457 time to create 1 rle with old method : 0.001978158950805664 time for calcul the mask position with numpy : 0.02168130874633789 nb_pixel_total : 186389 time to create 1 rle with new method : 0.3315134048461914 time for calcul the mask position with numpy : 0.020978450775146484 nb_pixel_total : 10743 time to create 1 rle with old method : 0.014068603515625 time for calcul the mask position with numpy : 0.021554946899414062 nb_pixel_total : 38602 time to create 1 rle with old method : 0.049523353576660156 time for calcul the mask position with numpy : 0.0213162899017334 nb_pixel_total : 6934 time to create 1 rle with old method : 0.008854389190673828 time for calcul the mask position with numpy : 0.022006988525390625 nb_pixel_total : 194637 time to create 1 rle with new method : 0.36379337310791016 time for calcul the mask position with numpy : 0.02160811424255371 nb_pixel_total : 10972 time to create 1 rle with old method : 0.0139007568359375 time for calcul the mask position with numpy : 0.02207779884338379 nb_pixel_total : 132508 time to create 1 rle with old method : 0.1626603603363037 create new chi : 2.771761417388916 time to delete rle : 0.0013194084167480469 batch 1 Loaded 20 chid ids of type : 3726 Number RLEs to save : 14454 TO DO : save crop sub photo not yet done ! save time : 1.8669450283050537 nb_obj : 18 nb_hashtags : 4 time to prepare the origin masks : 8.275467872619629 time for calcul the mask position with numpy : 0.12117528915405273 nb_pixel_total : 6121507 time to create 1 rle with new method : 0.4504122734069824 time for calcul the mask position with numpy : 0.019819974899291992 nb_pixel_total : 66280 time to create 1 rle with old method : 0.07139301300048828 time for calcul the mask position with numpy : 0.0196840763092041 nb_pixel_total : 15115 time to create 1 rle with old method : 0.018774986267089844 time for calcul the mask position with numpy : 0.019724130630493164 nb_pixel_total : 55996 time to create 1 rle with old method : 0.06226992607116699 time for calcul the mask position with numpy : 0.0194547176361084 nb_pixel_total : 5330 time to create 1 rle with old method : 0.006858110427856445 time for calcul the mask position with numpy : 0.01906609535217285 nb_pixel_total : 38550 time to create 1 rle with old method : 0.04634499549865723 time for calcul the mask position with numpy : 0.01924753189086914 nb_pixel_total : 2745 time to create 1 rle with old method : 0.003436565399169922 time for calcul the mask position with numpy : 0.019461393356323242 nb_pixel_total : 9398 time to create 1 rle with old method : 0.011808633804321289 time for calcul the mask position with numpy : 0.019793987274169922 nb_pixel_total : 107934 time to create 1 rle with old method : 0.11090087890625 time for calcul the mask position with numpy : 0.020464181900024414 nb_pixel_total : 115994 time to create 1 rle with old method : 0.1168513298034668 time for calcul the mask position with numpy : 0.01964879035949707 nb_pixel_total : 1448 time to create 1 rle with old method : 0.0018100738525390625 time for calcul the mask position with numpy : 0.019831180572509766 nb_pixel_total : 3310 time to create 1 rle with old method : 0.004193305969238281 time for calcul the mask position with numpy : 0.02048206329345703 nb_pixel_total : 39792 time to create 1 rle with old method : 0.04875993728637695 time for calcul the mask position with numpy : 0.019624948501586914 nb_pixel_total : 20907 time to create 1 rle with old method : 0.025166749954223633 time for calcul the mask position with numpy : 0.020519733428955078 nb_pixel_total : 65152 time to create 1 rle with old method : 0.07028818130493164 time for calcul the mask position with numpy : 0.01997518539428711 nb_pixel_total : 30007 time to create 1 rle with old method : 0.032517194747924805 time for calcul the mask position with numpy : 0.021225452423095703 nb_pixel_total : 135145 time to create 1 rle with old method : 0.1358780860900879 time for calcul the mask position with numpy : 0.02089405059814453 nb_pixel_total : 148620 time to create 1 rle with old method : 0.18300199508666992 time for calcul the mask position with numpy : 0.023900747299194336 nb_pixel_total : 67010 time to create 1 rle with old method : 0.0839226245880127 create new chi : 1.9982807636260986 time to delete rle : 0.0014100074768066406 batch 1 Loaded 19 chid ids of type : 3726 Number RLEs to save : 13941 TO DO : save crop sub photo not yet done ! save time : 2.02964186668396 nb_obj : 19 nb_hashtags : 5 time to prepare the origin masks : 8.334240913391113 time for calcul the mask position with numpy : 0.08371710777282715 nb_pixel_total : 4470007 time to create 1 rle with new method : 0.736102819442749 time for calcul the mask position with numpy : 0.027688026428222656 nb_pixel_total : 198639 time to create 1 rle with new method : 0.20929479598999023 time for calcul the mask position with numpy : 0.024486303329467773 nb_pixel_total : 250138 time to create 1 rle with new method : 0.21718835830688477 time for calcul the mask position with numpy : 0.021029949188232422 nb_pixel_total : 58784 time to create 1 rle with old method : 0.07080864906311035 time for calcul the mask position with numpy : 0.02063274383544922 nb_pixel_total : 4863 time to create 1 rle with old method : 0.006520271301269531 time for calcul the mask position with numpy : 0.029521942138671875 nb_pixel_total : 1271782 time to create 1 rle with new method : 0.2672312259674072 time for calcul the mask position with numpy : 0.02051711082458496 nb_pixel_total : 213 time to create 1 rle with old method : 0.00046825408935546875 time for calcul the mask position with numpy : 0.02039504051208496 nb_pixel_total : 14437 time to create 1 rle with old method : 0.018742799758911133 time for calcul the mask position with numpy : 0.02086663246154785 nb_pixel_total : 8214 time to create 1 rle with old method : 0.010824441909790039 time for calcul the mask position with numpy : 0.02187800407409668 nb_pixel_total : 80796 time to create 1 rle with old method : 0.10242247581481934 time for calcul the mask position with numpy : 0.021970033645629883 nb_pixel_total : 124819 time to create 1 rle with old method : 0.13836002349853516 time for calcul the mask position with numpy : 0.020864009857177734 nb_pixel_total : 1526 time to create 1 rle with old method : 0.0024857521057128906 time for calcul the mask position with numpy : 0.021274328231811523 nb_pixel_total : 821 time to create 1 rle with old method : 0.0022895336151123047 time for calcul the mask position with numpy : 0.023807764053344727 nb_pixel_total : 340048 time to create 1 rle with new method : 0.2252354621887207 time for calcul the mask position with numpy : 0.024379253387451172 nb_pixel_total : 7582 time to create 1 rle with old method : 0.010360002517700195 time for calcul the mask position with numpy : 0.02158188819885254 nb_pixel_total : 703 time to create 1 rle with old method : 0.0013077259063720703 time for calcul the mask position with numpy : 0.02260112762451172 nb_pixel_total : 74995 time to create 1 rle with old method : 0.09442877769470215 time for calcul the mask position with numpy : 0.024036407470703125 nb_pixel_total : 11100 time to create 1 rle with old method : 0.013707876205444336 time for calcul the mask position with numpy : 0.02119302749633789 nb_pixel_total : 124719 time to create 1 rle with old method : 0.15503144264221191 time for calcul the mask position with numpy : 0.0209505558013916 nb_pixel_total : 6054 time to create 1 rle with old method : 0.007862091064453125 create new chi : 2.9044456481933594 time to delete rle : 0.0020661354064941406 batch 1 Loaded 21 chid ids of type : 3726 Number RLEs to save : 18529 TO DO : save crop sub photo not yet done ! save time : 0.9930262565612793 nb_obj : 15 nb_hashtags : 7 time to prepare the origin masks : 6.5726213455200195 time for calcul the mask position with numpy : 0.25522828102111816 nb_pixel_total : 5134140 time to create 1 rle with new method : 0.6410131454467773 time for calcul the mask position with numpy : 0.020968198776245117 nb_pixel_total : 16810 time to create 1 rle with old method : 0.021808624267578125 time for calcul the mask position with numpy : 0.021344423294067383 nb_pixel_total : 19969 time to create 1 rle with old method : 0.026053428649902344 time for calcul the mask position with numpy : 0.020694255828857422 nb_pixel_total : 15242 time to create 1 rle with old method : 0.03060603141784668 time for calcul the mask position with numpy : 0.02146458625793457 nb_pixel_total : 11808 time to create 1 rle with old method : 0.014343500137329102 time for calcul the mask position with numpy : 0.020955562591552734 nb_pixel_total : 31505 time to create 1 rle with old method : 0.04014778137207031 time for calcul the mask position with numpy : 0.022650480270385742 nb_pixel_total : 141694 time to create 1 rle with old method : 0.15645909309387207 time for calcul the mask position with numpy : 0.0257265567779541 nb_pixel_total : 489309 time to create 1 rle with new method : 0.42927098274230957 time for calcul the mask position with numpy : 0.022863388061523438 nb_pixel_total : 198877 time to create 1 rle with new method : 0.4189116954803467 time for calcul the mask position with numpy : 0.025791406631469727 nb_pixel_total : 706570 time to create 1 rle with new method : 0.35847902297973633 time for calcul the mask position with numpy : 0.022072553634643555 nb_pixel_total : 36377 time to create 1 rle with old method : 0.03914332389831543 time for calcul the mask position with numpy : 0.02052164077758789 nb_pixel_total : 40487 time to create 1 rle with old method : 0.046975135803222656 time for calcul the mask position with numpy : 0.020139217376708984 nb_pixel_total : 5477 time to create 1 rle with old method : 0.0062923431396484375 time for calcul the mask position with numpy : 0.02081441879272461 nb_pixel_total : 5828 time to create 1 rle with old method : 0.006894588470458984 time for calcul the mask position with numpy : 0.02151012420654297 nb_pixel_total : 10243 time to create 1 rle with old method : 0.012958049774169922 time for calcul the mask position with numpy : 0.021834611892700195 nb_pixel_total : 185904 time to create 1 rle with new method : 0.37486696243286133 create new chi : 3.3461852073669434 time to delete rle : 0.0017578601837158203 batch 1 Loaded 16 chid ids of type : 3726 Number RLEs to save : 16421 TO DO : save crop sub photo not yet done ! save time : 0.8968527317047119 nb_obj : 8 nb_hashtags : 4 time to prepare the origin masks : 2.5507328510284424 time for calcul the mask position with numpy : 0.2579076290130615 nb_pixel_total : 6751910 time to create 1 rle with new method : 0.5261397361755371 time for calcul the mask position with numpy : 0.02156352996826172 nb_pixel_total : 73934 time to create 1 rle with old method : 0.08961844444274902 time for calcul the mask position with numpy : 0.020730972290039062 nb_pixel_total : 8731 time to create 1 rle with old method : 0.010779142379760742 time for calcul the mask position with numpy : 0.02098870277404785 nb_pixel_total : 102366 time to create 1 rle with old method : 0.1232600212097168 time for calcul the mask position with numpy : 0.02251267433166504 nb_pixel_total : 12222 time to create 1 rle with old method : 0.01484227180480957 time for calcul the mask position with numpy : 0.022975683212280273 nb_pixel_total : 9034 time to create 1 rle with old method : 0.013057947158813477 time for calcul the mask position with numpy : 0.02191638946533203 nb_pixel_total : 61336 time to create 1 rle with old method : 0.08152031898498535 time for calcul the mask position with numpy : 0.026352405548095703 nb_pixel_total : 7352 time to create 1 rle with old method : 0.010240316390991211 time for calcul the mask position with numpy : 0.02395319938659668 nb_pixel_total : 23355 time to create 1 rle with old method : 0.029993772506713867 create new chi : 1.36995530128479 time to delete rle : 0.0007755756378173828 batch 1 Loaded 9 chid ids of type : 3726 Number RLEs to save : 5510 TO DO : save crop sub photo not yet done ! save time : 0.3301241397857666 nb_obj : 15 nb_hashtags : 5 time to prepare the origin masks : 6.510047435760498 time for calcul the mask position with numpy : 0.23684048652648926 nb_pixel_total : 6654033 time to create 1 rle with new method : 0.299910306930542 time for calcul the mask position with numpy : 0.02156972885131836 nb_pixel_total : 14161 time to create 1 rle with old method : 0.018654823303222656 time for calcul the mask position with numpy : 0.02236175537109375 nb_pixel_total : 41046 time to create 1 rle with old method : 0.050679922103881836 time for calcul the mask position with numpy : 0.02032327651977539 nb_pixel_total : 9990 time to create 1 rle with old method : 0.013234615325927734 time for calcul the mask position with numpy : 0.02118062973022461 nb_pixel_total : 1285 time to create 1 rle with old method : 0.0018947124481201172 time for calcul the mask position with numpy : 0.02108621597290039 nb_pixel_total : 5909 time to create 1 rle with old method : 0.008010149002075195 time for calcul the mask position with numpy : 0.020473957061767578 nb_pixel_total : 4883 time to create 1 rle with old method : 0.006484508514404297 time for calcul the mask position with numpy : 0.02073812484741211 nb_pixel_total : 67602 time to create 1 rle with old method : 0.10322880744934082 time for calcul the mask position with numpy : 0.02241659164428711 nb_pixel_total : 76285 time to create 1 rle with old method : 0.09775066375732422 time for calcul the mask position with numpy : 0.0205533504486084 nb_pixel_total : 4014 time to create 1 rle with old method : 0.0054171085357666016 time for calcul the mask position with numpy : 0.02109980583190918 nb_pixel_total : 18213 time to create 1 rle with old method : 0.02290630340576172 time for calcul the mask position with numpy : 0.02135491371154785 nb_pixel_total : 138607 time to create 1 rle with old method : 0.1490342617034912 time for calcul the mask position with numpy : 0.020580291748046875 nb_pixel_total : 4062 time to create 1 rle with old method : 0.005473613739013672 time for calcul the mask position with numpy : 0.021104812622070312 nb_pixel_total : 7469 time to create 1 rle with old method : 0.00992584228515625 time for calcul the mask position with numpy : 0.020514726638793945 nb_pixel_total : 65 time to create 1 rle with old method : 0.0001850128173828125 time for calcul the mask position with numpy : 0.020605087280273438 nb_pixel_total : 2616 time to create 1 rle with old method : 0.0035126209259033203 create new chi : 1.3876924514770508 time to delete rle : 0.0009388923645019531 batch 1 Loaded 16 chid ids of type : 3726 Number RLEs to save : 8014 TO DO : save crop sub photo not yet done ! save time : 0.45231008529663086 map_output_result : {1334497458: (0.0, 'Should be the crop_list due to order', 0.0), 1334497454: (0.0, 'Should be the crop_list due to order', 0.0), 1334497450: (0.0, 'Should be the crop_list due to order', 0.0), 1334497447: (0.0, 'Should be the crop_list due to order', 0.0), 1334497444: (0.0, 'Should be the crop_list due to order', 0.0), 1334497441: (0.0, 'Should be the crop_list due to order', 0.0), 1334497438: (0.0, 'Should be the crop_list due to order', 0.0), 1334497434: (0.0, 'Should be the crop_list due to order', 0.0), 1334497411: (0.0, 'Should be the crop_list due to order', 0.0), 1334497400: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1334497458, 1334497454, 1334497450, 1334497447, 1334497444, 1334497441, 1334497438, 1334497434, 1334497411, 1334497400] Looping around the photos to save general results len do output : 10 /1334497458.Didn't retrieve data . /1334497454.Didn't retrieve data . /1334497450.Didn't retrieve data . /1334497447.Didn't retrieve data . /1334497444.Didn't retrieve data . /1334497441.Didn't retrieve data . /1334497438.Didn't retrieve data . /1334497434.Didn't retrieve data . /1334497411.Didn't retrieve data . /1334497400.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497458', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497454', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497450', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497447', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497444', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497441', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497438', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497434', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497411', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497400', None, None, None, None, None, '2551093') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013058662414550781 save_final save missing photos in datou_result : time spend for datou_step_exec : 105.93463182449341 time spend to save output : 0.01721811294555664 total time spend for step 5 : 105.95184993743896 step6:crop_condition Wed Feb 5 13:00:33 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 181 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738756836_3701782 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.635972261428833 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! About to upload 7 photos upload in portfolio : 4869462 init cache_photo without model_param we have 7 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738756842_3701782 we have uploaded 7 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.7709617614746094 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738756847_3701782 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.3183486461639404 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738756849_3701782 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.8281915187835693 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! About to upload 5 photos upload in portfolio : 4869462 init cache_photo without model_param we have 5 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738756852_3701782 we have uploaded 5 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.4681870937347412 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738756855_3701782 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6276450157165527 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738756857_3701782 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.5897722244262695 we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1334497458, 1334497454, 1334497450, 1334497447, 1334497444, 1334497441, 1334497438, 1334497434, 1334497411, 1334497400] Looping around the photos to save general results len do output : 26 /1334919622Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334919623Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334919624Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334919625Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334919626Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334919627Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334919632Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334919633Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334919634Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334919635Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334919636Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334919637Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334919638Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334919641Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334919642Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334919643Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334919644Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334919645Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334919646Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334919649Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334919650Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334919651Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334919652Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334919653Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334919655Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1334919657Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497458', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497454', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497450', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497447', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497444', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497441', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497438', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497434', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497411', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497400', None, None, None, None, None, '2551093') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 88 time used for this insertion : 0.031156539916992188 save_final save missing photos in datou_result : time spend for datou_step_exec : 25.591644287109375 time spend to save output : 0.04318094253540039 total time spend for step 6 : 25.634825229644775 step7:ventilate_hashtags_in_portfolio Wed Feb 5 13:00:58 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20254669 get user id for portfolio 20254669 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254669 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('flou','Teint_Dans_La_Masse','autre_refus','kraft','environnement','Carton_brun','cartonnette','Carton_gris','plastique','metal','papier','mal_croppe')) AND mptpi.`min_score`=0.5 To do To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254669 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('flou','Teint_Dans_La_Masse','autre_refus','kraft','environnement','Carton_brun','cartonnette','Carton_gris','plastique','metal','papier','mal_croppe')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20255897,20255898,20255899,20255900,20255901,20255902,20255903,20255904,20255905,20255906,20255907,20255908?tags=environnement,metal,cartonnette,mal_croppe,Carton_gris,flou,Carton_brun,kraft,plastique,Teint_Dans_La_Masse,autre_refus,papier Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1334497458, 1334497454, 1334497450, 1334497447, 1334497444, 1334497441, 1334497438, 1334497434, 1334497411, 1334497400] Looping around the photos to save general results len do output : 1 /20254669. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497458', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497454', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497450', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497447', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497444', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497441', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497438', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497434', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497411', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497400', None, None, None, None, None, '2551093') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.015005111694335938 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.5684447288513184 time spend to save output : 0.015332698822021484 total time spend for step 7 : 0.5837774276733398 step8:final Wed Feb 5 13:00:59 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1334497458: ('0.07156994082038416',), 1334497454: ('0.07156994082038416',), 1334497450: ('0.07156994082038416',), 1334497447: ('0.07156994082038416',), 1334497444: ('0.07156994082038416',), 1334497441: ('0.07156994082038416',), 1334497438: ('0.07156994082038416',), 1334497434: ('0.07156994082038416',), 1334497411: ('0.07156994082038416',), 1334497400: ('0.07156994082038416',)} new output for save of step final : {1334497458: ('0.07156994082038416',), 1334497454: ('0.07156994082038416',), 1334497450: ('0.07156994082038416',), 1334497447: ('0.07156994082038416',), 1334497444: ('0.07156994082038416',), 1334497441: ('0.07156994082038416',), 1334497438: ('0.07156994082038416',), 1334497434: ('0.07156994082038416',), 1334497411: ('0.07156994082038416',), 1334497400: ('0.07156994082038416',)} [1334497458, 1334497454, 1334497450, 1334497447, 1334497444, 1334497441, 1334497438, 1334497434, 1334497411, 1334497400] Looping around the photos to save general results len do output : 10 /1334497458.Didn't retrieve data . /1334497454.Didn't retrieve data . /1334497450.Didn't retrieve data . /1334497447.Didn't retrieve data . /1334497444.Didn't retrieve data . /1334497441.Didn't retrieve data . /1334497438.Didn't retrieve data . /1334497434.Didn't retrieve data . /1334497411.Didn't retrieve data . /1334497400.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497458', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497454', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497450', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497447', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497444', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497441', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497438', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497434', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497411', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497400', None, None, None, None, None, '2551093') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.0135345458984375 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.16069388389587402 time spend to save output : 0.014255285263061523 total time spend for step 8 : 0.17494916915893555 step9:velours_tree Wed Feb 5 13:00:59 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 1.0977470874786377 time spend to save output : 9.34600830078125e-05 total time spend for step 9 : 1.0978405475616455 step10:send_mail_cod Wed Feb 5 13:01:00 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20254669_05-02-2025_13_01_00.pdf 20255898 change filename to text .imagette202558981738756860 20255899 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202558991738756860 20255900 imagette202559001738756862 20255902 imagette202559021738756862 20255904 change filename to text .change filename to text .change filename to text .change filename to text .imagette202559041738756862 20255905 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202559051738756862 20255907 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202559071738756863 20255901 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202559011738756864 20255903 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202559031738756866 20255906 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette202559061738756867 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20254669 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20255897,20255898,20255899,20255900,20255901,20255902,20255903,20255904,20255905,20255906,20255907,20255908?tags=environnement,metal,cartonnette,mal_croppe,Carton_gris,flou,Carton_brun,kraft,plastique,Teint_Dans_La_Masse,autre_refus,papier your option no_mail is active, we will not send the real mail to your client args[1334497458] : ('0.07156994082038416',) no score found for photo 1334497458 We are sending mail with results at report@fotonower.com args[1334497454] : ('0.07156994082038416',) no score found for photo 1334497454 We are sending mail with results at report@fotonower.com args[1334497450] : ('0.07156994082038416',) no score found for photo 1334497450 We are sending mail with results at report@fotonower.com args[1334497447] : ('0.07156994082038416',) no score found for photo 1334497447 We are sending mail with results at report@fotonower.com args[1334497444] : ('0.07156994082038416',) no score found for photo 1334497444 We are sending mail with results at report@fotonower.com args[1334497441] : ('0.07156994082038416',) no score found for photo 1334497441 We are sending mail with results at report@fotonower.com args[1334497438] : ('0.07156994082038416',) no score found for photo 1334497438 We are sending mail with results at report@fotonower.com args[1334497434] : ('0.07156994082038416',) no score found for photo 1334497434 We are sending mail with results at report@fotonower.com args[1334497411] : ('0.07156994082038416',) no score found for photo 1334497411 We are sending mail with results at report@fotonower.com args[1334497400] : ('0.07156994082038416',) no score found for photo 1334497400 We are sending mail with results at report@fotonower.com refus_total : 0.07156994082038416 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20254669 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1334496996,1334696715,1334497467,1334497463,1334497458,1334497454,1334497450,1334711697,1334497444,1334497441,1334497438,1334711699,1334497411,1334497400,1334497358,1334497355,1334497331,1334497302,1334496991,1334497008) Found this number of photos: 20 begin to download photo : 1334496996 begin to download photo : 1334497454 begin to download photo : 1334497438 begin to download photo : 1334497355 download finish for photo 1334497454 begin to download photo : 1334497450 download finish for photo 1334497355 begin to download photo : 1334497331 download finish for photo 1334497438 begin to download photo : 1334711699 download finish for photo 1334496996 begin to download photo : 1334696715 download finish for photo 1334497450 begin to download photo : 1334711697 download finish for photo 1334711699 begin to download photo : 1334497411 download finish for photo 1334497331 begin to download photo : 1334497302 download finish for photo 1334696715 begin to download photo : 1334497467 download finish for photo 1334711697 begin to download photo : 1334497444 download finish for photo 1334497411 begin to download photo : 1334497400 download finish for photo 1334497302 begin to download photo : 1334496991 download finish for photo 1334497467 begin to download photo : 1334497463 download finish for photo 1334497444 begin to download photo : 1334497441 download finish for photo 1334497400 begin to download photo : 1334497358 download finish for photo 1334496991 begin to download photo : 1334497008 download finish for photo 1334497463 begin to download photo : 1334497458 download finish for photo 1334497441 download finish for photo 1334497008 download finish for photo 1334497358 download finish for photo 1334497458 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254669_05-02-2025_13_01_00.pdf results_Auto_P20254669_05-02-2025_13_01_00.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254669_05-02-2025_13_01_00.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20254669','results_Auto_P20254669_05-02-2025_13_01_00.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254669_05-02-2025_13_01_00.pdf','pdf','','1.34','0.07156994082038416') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1334497458, 1334497454, 1334497450, 1334497447, 1334497444, 1334497441, 1334497438, 1334497434, 1334497411, 1334497400] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497458', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497454', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497450', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497447', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497444', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497441', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497438', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497434', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497411', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497400', None, None, None, None, None, '2551093') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.01331472396850586 save_final save missing photos in datou_result : time spend for datou_step_exec : 12.858283758163452 time spend to save output : 0.013581514358520508 total time spend for step 10 : 12.871865272521973 step11:split_time_score Wed Feb 5 13:01:13 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('09', 23), ('10', 5)) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 04022025 20254669 Nombre de photos uploadées : 28 / 23040 (0%) 04022025 20254669 Nombre de photos taguées (types de déchets): 0 / 28 (0%) 04022025 20254669 Nombre de photos taguées (volume) : 0 / 28 (0%) elapsed_time : load_data_split_time_score 2.384185791015625e-06 elapsed_time : order_list_meta_photo_and_scores 6.198883056640625e-06 ???????????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.001893758773803711 elapsed_time : insert_dashboard_record_day_entry 0.022042512893676758 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254648 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254650 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254653 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254655 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254657 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254660 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254661 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254663 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254664 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254666 order by id desc limit 1 Qualite : 0.03827887555059262 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20208806_04-02-2025_12_29_58.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20208806 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20208806 AND mptpi.`type`=4207 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254668 order by id desc limit 1 Qualite : 0.07156960103454109 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254669_05-02-2025_13_01_00.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254669 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254669 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254671 order by id desc limit 1 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254672_05-02-2025_12_22_04.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254672 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254674 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254675 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254688 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254690 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254691 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254692 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254693 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254694 order by id desc limit 1 Qualite : 0.29069025611709665 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254695_05-02-2025_12_46_09.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254695 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11548 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11556 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11557 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11552 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11552 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11555 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11555 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11580 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11580 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11551 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11550 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11550 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11559 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11551 doesn't seem to be define in the database( WARNING : type of input 3 of step 11550 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11548 doesn't seem to be define in the database( WARNING : type of input 2 of step 11552 doesn't seem to be define in the database( WARNING : output 1 of step 11548 have datatype=2 whereas input 1 of step 11555 have datatype=7 WARNING : type of output 2 of step 11555 doesn't seem to be define in the database( WARNING : type of input 1 of step 11549 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11551 have datatype=10 whereas input 3 of step 11558 have datatype=6 WARNING : type of input 2 of step 11580 doesn't seem to be define in the database( WARNING : output 1 of step 11549 have datatype=7 whereas input 2 of step 11580 have datatype=None WARNING : type of output 3 of step 11580 doesn't seem to be define in the database( WARNING : type of input 1 of step 11551 doesn't seem to be define in the database( WARNING : output 0 of step 11551 have datatype=10 whereas input 0 of step 11586 have datatype=18 WARNING : type of input 5 of step 11558 doesn't seem to be define in the database( WARNING : output 0 of step 11586 have datatype=11 whereas input 5 of step 11558 have datatype=None WARNING : type of output 1 of step 11556 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : type of output 1 of step 11557 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : output 0 of step 11555 have datatype=1 whereas input 0 of step 11549 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254695 AND mptpi.`type`=4200 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254696 order by id desc limit 1 Qualite : 0.026493133846067357 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254697 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11560 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11567 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11567 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11563 crop_condition is not consistent : 4 used against 2 in the step definition ! Step 11563 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11564 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11564 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11573 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11573 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11566 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11566 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 1 of step 11560 have datatype=2 whereas input 1 of step 11564 have datatype=7 WARNING : type of output 2 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11565 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11567 doesn't seem to be define in the database( WARNING : type of input 3 of step 11563 doesn't seem to be define in the database( WARNING : type of output 3 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11568 doesn't seem to be define in the database( WARNING : type of output 1 of step 11568 doesn't seem to be define in the database( WARNING : type of input 3 of step 11566 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11570 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11569 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11570 doesn't seem to be define in the database( WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11569 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11568 have datatype=10 whereas input 3 of step 11571 have datatype=6 WARNING : type of input 2 of step 11573 doesn't seem to be define in the database( WARNING : output 1 of step 11565 have datatype=7 whereas input 2 of step 11573 have datatype=None WARNING : type of output 3 of step 11573 doesn't seem to be define in the database( WARNING : type of input 3 of step 11568 doesn't seem to be define in the database( WARNING : output 0 of step 11568 have datatype=10 whereas input 0 of step 11587 have datatype=18 WARNING : type of input 5 of step 11571 doesn't seem to be define in the database( WARNING : output 0 of step 11587 have datatype=11 whereas input 5 of step 11571 have datatype=None WARNING : output 0 of step 11564 have datatype=1 whereas input 0 of step 11565 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254697 AND mptpi.`type`=3327 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254698 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254699 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254700 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254701 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254716 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254727 order by id desc limit 1 Qualite : 0.21864209909553048 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254728_05-02-2025_12_49_49.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254728 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254728 AND mptpi.`type`=3594 To do Qualite : 0.21847787515679665 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254729_05-02-2025_12_41_35.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254729 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254729 AND mptpi.`type`=3594 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254730 order by id desc limit 1 Qualite : 0.17446064779355028 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254731_05-02-2025_12_32_03.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254731 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254731 AND mptpi.`type`=3594 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254732 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254733 order by id desc limit 1 Qualite : 0.5901434799670006 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254734_05-02-2025_12_42_43.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254734 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11548 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11556 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11557 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11552 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11552 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11555 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11555 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11580 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11580 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11551 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11550 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11550 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11559 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11551 doesn't seem to be define in the database( WARNING : type of input 3 of step 11550 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11548 doesn't seem to be define in the database( WARNING : type of input 2 of step 11552 doesn't seem to be define in the database( WARNING : output 1 of step 11548 have datatype=2 whereas input 1 of step 11555 have datatype=7 WARNING : type of output 2 of step 11555 doesn't seem to be define in the database( WARNING : type of input 1 of step 11549 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11551 have datatype=10 whereas input 3 of step 11558 have datatype=6 WARNING : type of input 2 of step 11580 doesn't seem to be define in the database( WARNING : output 1 of step 11549 have datatype=7 whereas input 2 of step 11580 have datatype=None WARNING : type of output 3 of step 11580 doesn't seem to be define in the database( WARNING : type of input 1 of step 11551 doesn't seem to be define in the database( WARNING : output 0 of step 11551 have datatype=10 whereas input 0 of step 11586 have datatype=18 WARNING : type of input 5 of step 11558 doesn't seem to be define in the database( WARNING : output 0 of step 11586 have datatype=11 whereas input 5 of step 11558 have datatype=None WARNING : type of output 1 of step 11556 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : type of output 1 of step 11557 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : output 0 of step 11555 have datatype=1 whereas input 0 of step 11549 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254734 AND mptpi.`type`=4200 To do Qualite : 0.1455004789653495 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254735_05-02-2025_12_58_16.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254735 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11560 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11567 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11567 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11563 crop_condition is not consistent : 4 used against 2 in the step definition ! Step 11563 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11564 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11564 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11573 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11573 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11566 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11566 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 1 of step 11560 have datatype=2 whereas input 1 of step 11564 have datatype=7 WARNING : type of output 2 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11565 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11567 doesn't seem to be define in the database( WARNING : type of input 3 of step 11563 doesn't seem to be define in the database( WARNING : type of output 3 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11568 doesn't seem to be define in the database( WARNING : type of output 1 of step 11568 doesn't seem to be define in the database( WARNING : type of input 3 of step 11566 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11570 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11569 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11570 doesn't seem to be define in the database( WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11569 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11568 have datatype=10 whereas input 3 of step 11571 have datatype=6 WARNING : type of input 2 of step 11573 doesn't seem to be define in the database( WARNING : output 1 of step 11565 have datatype=7 whereas input 2 of step 11573 have datatype=None WARNING : type of output 3 of step 11573 doesn't seem to be define in the database( WARNING : type of input 3 of step 11568 doesn't seem to be define in the database( WARNING : output 0 of step 11568 have datatype=10 whereas input 0 of step 11587 have datatype=18 WARNING : type of input 5 of step 11571 doesn't seem to be define in the database( WARNING : output 0 of step 11587 have datatype=11 whereas input 5 of step 11571 have datatype=None WARNING : output 0 of step 11564 have datatype=1 whereas input 0 of step 11565 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254735 AND mptpi.`type`=3327 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254736 order by id desc limit 1 Qualite : 0.08123124611481618 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254737_05-02-2025_12_58_44.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254737 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254737 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254738 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254739 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254740 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254741 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254742 order by id desc limit 1 Qualite : 0.09993071763572346 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254767_05-02-2025_11_50_14.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254767 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254767 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254768 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254769 order by id desc limit 1 Qualite : 0.2029836809337933 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254770_05-02-2025_11_49_33.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254770 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254770 AND mptpi.`type`=3594 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254771 order by id desc limit 1 Qualite : 0.07408981285512192 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254772_05-02-2025_11_59_52.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254772 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254772 AND mptpi.`type`=4207 To do Qualite : 0.09660837363298684 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20254773_05-02-2025_12_36_26.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20254773 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20254773 AND mptpi.`type`=4207 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'04022025': {'nb_upload': 28, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1334497458, 1334497454, 1334497450, 1334497447, 1334497444, 1334497441, 1334497438, 1334497434, 1334497411, 1334497400] Looping around the photos to save general results len do output : 1 /20254669Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497458', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497454', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497450', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497447', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497444', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497441', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497438', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497434', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497411', None, None, None, None, None, '2551093') ('3459', None, None, None, None, None, None, None, '2551093') ('3459', None, '1334497400', None, None, None, None, None, '2551093') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.020259857177734375 save_final save missing photos in datou_result : time spend for datou_step_exec : 30.788475036621094 time spend to save output : 0.02096247673034668 total time spend for step 11 : 30.80943751335144 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 162.73user 102.29system 5:21.93elapsed 82%CPU (0avgtext+0avgdata 3737920maxresident)k 3283632inputs+164608outputs (29147major+12025693minor)pagefaults 0swaps