python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 1427269 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 20>10 please execute split_portfolio.py -i 21164371 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2650228'] with mtr_portfolio_ids : ['21164371'] and first list_photo_ids : [] new path : /proc/1427269/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 1.7532525062561035 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Sat Mar 8 00:18:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5178 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-03-08 00:18:33.056554: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-08 00:18:33.091089: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-08 00:18:33.092810: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f17d0000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-08 00:18:33.092833: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-08 00:18:33.096380: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-08 00:18:33.252302: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x9e89c90 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-08 00:18:33.252373: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-08 00:18:33.253624: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-08 00:18:33.254079: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-08 00:18:33.257152: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-08 00:18:33.260192: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-08 00:18:33.260968: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-08 00:18:33.264370: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-08 00:18:33.266071: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-08 00:18:33.272164: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-08 00:18:33.273453: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-08 00:18:33.273549: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-08 00:18:33.274191: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-08 00:18:33.274208: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-08 00:18:33.274217: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-08 00:18:33.275630: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4725 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-03-08 00:18:33.589013: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-08 00:18:33.589134: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-08 00:18:33.589156: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-08 00:18:33.589175: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-08 00:18:33.589192: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-08 00:18:33.589210: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-08 00:18:33.589228: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-08 00:18:33.589246: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-08 00:18:33.590445: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-08 00:18:33.591575: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-08 00:18:33.591619: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-08 00:18:33.591638: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-08 00:18:33.591655: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-08 00:18:33.591672: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-08 00:18:33.591689: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-08 00:18:33.591709: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-08 00:18:33.591727: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-08 00:18:33.592929: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-08 00:18:33.592963: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-08 00:18:33.592974: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-08 00:18:33.592984: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-08 00:18:33.594192: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4725 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-03-08 00:18:45.920928: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-08 00:18:46.143225: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-08 00:18:47.962261: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 19 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 20 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.02358 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 17 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 8 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 150.92312 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 23 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 19 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 12 Detection mask done ! Trying to reset tf kernel 1427692 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 297 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5178 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.0006520748138427734 nb_pixel_total : 14468 time to create 1 rle with old method : 0.02023911476135254 length of segment : 170 time for calcul the mask position with numpy : 0.0013039112091064453 nb_pixel_total : 50591 time to create 1 rle with old method : 0.06497311592102051 length of segment : 380 time for calcul the mask position with numpy : 0.00027871131896972656 nb_pixel_total : 8490 time to create 1 rle with old method : 0.011210918426513672 length of segment : 164 time for calcul the mask position with numpy : 0.0016639232635498047 nb_pixel_total : 75232 time to create 1 rle with old method : 0.09088492393493652 length of segment : 389 time for calcul the mask position with numpy : 0.0004076957702636719 nb_pixel_total : 16792 time to create 1 rle with old method : 0.03329205513000488 length of segment : 163 time for calcul the mask position with numpy : 0.0004508495330810547 nb_pixel_total : 16257 time to create 1 rle with old method : 0.022216796875 length of segment : 251 time for calcul the mask position with numpy : 0.0014929771423339844 nb_pixel_total : 32779 time to create 1 rle with old method : 0.09078788757324219 length of segment : 465 time for calcul the mask position with numpy : 0.00043463706970214844 nb_pixel_total : 12814 time to create 1 rle with old method : 0.016785860061645508 length of segment : 382 time for calcul the mask position with numpy : 0.00016546249389648438 nb_pixel_total : 2301 time to create 1 rle with old method : 0.0030837059020996094 length of segment : 89 time for calcul the mask position with numpy : 8.559226989746094e-05 nb_pixel_total : 2471 time to create 1 rle with old method : 0.003328084945678711 length of segment : 56 time for calcul the mask position with numpy : 0.00031447410583496094 nb_pixel_total : 16100 time to create 1 rle with old method : 0.020174264907836914 length of segment : 145 time for calcul the mask position with numpy : 0.0017707347869873047 nb_pixel_total : 61863 time to create 1 rle with old method : 0.07656216621398926 length of segment : 348 time for calcul the mask position with numpy : 0.024163246154785156 nb_pixel_total : 602929 time to create 1 rle with new method : 0.05550026893615723 length of segment : 1073 time for calcul the mask position with numpy : 0.0015256404876708984 nb_pixel_total : 105889 time to create 1 rle with old method : 0.1201639175415039 length of segment : 345 time for calcul the mask position with numpy : 0.0008819103240966797 nb_pixel_total : 28723 time to create 1 rle with old method : 0.05144762992858887 length of segment : 174 time for calcul the mask position with numpy : 0.0015149116516113281 nb_pixel_total : 85331 time to create 1 rle with old method : 0.11019396781921387 length of segment : 258 time for calcul the mask position with numpy : 0.0017881393432617188 nb_pixel_total : 89285 time to create 1 rle with old method : 0.1048133373260498 length of segment : 272 time for calcul the mask position with numpy : 0.00164794921875 nb_pixel_total : 45609 time to create 1 rle with old method : 0.058464765548706055 length of segment : 514 time for calcul the mask position with numpy : 0.0003170967102050781 nb_pixel_total : 5477 time to create 1 rle with old method : 0.0067861080169677734 length of segment : 77 time for calcul the mask position with numpy : 0.0015320777893066406 nb_pixel_total : 49628 time to create 1 rle with old method : 0.06297898292541504 length of segment : 213 time for calcul the mask position with numpy : 0.0002543926239013672 nb_pixel_total : 5194 time to create 1 rle with old method : 0.006972551345825195 length of segment : 59 time for calcul the mask position with numpy : 0.0012810230255126953 nb_pixel_total : 41888 time to create 1 rle with old method : 0.04972577095031738 length of segment : 225 time for calcul the mask position with numpy : 0.001486063003540039 nb_pixel_total : 33506 time to create 1 rle with old method : 0.038100481033325195 length of segment : 247 time for calcul the mask position with numpy : 0.003188610076904297 nb_pixel_total : 169073 time to create 1 rle with new method : 0.003710508346557617 length of segment : 461 time for calcul the mask position with numpy : 0.0009343624114990234 nb_pixel_total : 23997 time to create 1 rle with old method : 0.028725385665893555 length of segment : 287 time for calcul the mask position with numpy : 0.0009627342224121094 nb_pixel_total : 17607 time to create 1 rle with old method : 0.02127552032470703 length of segment : 327 time for calcul the mask position with numpy : 0.0010361671447753906 nb_pixel_total : 11631 time to create 1 rle with old method : 0.01518106460571289 length of segment : 190 time for calcul the mask position with numpy : 0.002986907958984375 nb_pixel_total : 166826 time to create 1 rle with new method : 0.006350278854370117 length of segment : 477 time for calcul the mask position with numpy : 0.0008172988891601562 nb_pixel_total : 25509 time to create 1 rle with old method : 0.030897140502929688 length of segment : 174 time for calcul the mask position with numpy : 0.0007979869842529297 nb_pixel_total : 34078 time to create 1 rle with old method : 0.0439455509185791 length of segment : 233 time for calcul the mask position with numpy : 0.0007076263427734375 nb_pixel_total : 17668 time to create 1 rle with old method : 0.0253903865814209 length of segment : 231 time for calcul the mask position with numpy : 0.0005402565002441406 nb_pixel_total : 19883 time to create 1 rle with old method : 0.03199934959411621 length of segment : 81 time for calcul the mask position with numpy : 0.0011320114135742188 nb_pixel_total : 33732 time to create 1 rle with old method : 0.04241800308227539 length of segment : 171 time for calcul the mask position with numpy : 0.0007984638214111328 nb_pixel_total : 28231 time to create 1 rle with old method : 0.03497672080993652 length of segment : 200 time for calcul the mask position with numpy : 0.15727519989013672 nb_pixel_total : 3911985 time to create 1 rle with new method : 0.6186943054199219 length of segment : 2455 time for calcul the mask position with numpy : 0.009733438491821289 nb_pixel_total : 438394 time to create 1 rle with new method : 0.17881107330322266 length of segment : 762 time for calcul the mask position with numpy : 0.00019884109497070312 nb_pixel_total : 2409 time to create 1 rle with old method : 0.0033004283905029297 length of segment : 57 time for calcul the mask position with numpy : 0.0006229877471923828 nb_pixel_total : 17094 time to create 1 rle with old method : 0.021784543991088867 length of segment : 186 time for calcul the mask position with numpy : 0.00042629241943359375 nb_pixel_total : 15231 time to create 1 rle with old method : 0.020641088485717773 length of segment : 138 time for calcul the mask position with numpy : 0.0001251697540283203 nb_pixel_total : 1161 time to create 1 rle with old method : 0.0015277862548828125 length of segment : 47 time for calcul the mask position with numpy : 0.006485462188720703 nb_pixel_total : 220706 time to create 1 rle with new method : 0.014792442321777344 length of segment : 707 time for calcul the mask position with numpy : 0.0031118392944335938 nb_pixel_total : 212300 time to create 1 rle with new method : 0.0073430538177490234 length of segment : 699 time for calcul the mask position with numpy : 0.002135038375854492 nb_pixel_total : 128167 time to create 1 rle with old method : 0.15848565101623535 length of segment : 393 time for calcul the mask position with numpy : 0.23449206352233887 nb_pixel_total : 4880117 time to create 1 rle with new method : 1.6068227291107178 length of segment : 4774 time for calcul the mask position with numpy : 0.0008940696716308594 nb_pixel_total : 38437 time to create 1 rle with old method : 0.04444265365600586 length of segment : 234 time for calcul the mask position with numpy : 0.019336462020874023 nb_pixel_total : 831870 time to create 1 rle with new method : 0.4541139602661133 length of segment : 2975 time for calcul the mask position with numpy : 0.0034818649291992188 nb_pixel_total : 212399 time to create 1 rle with new method : 0.010421037673950195 length of segment : 1090 time for calcul the mask position with numpy : 0.0032362937927246094 nb_pixel_total : 195095 time to create 1 rle with new method : 0.006084442138671875 length of segment : 606 time for calcul the mask position with numpy : 0.0020885467529296875 nb_pixel_total : 72811 time to create 1 rle with old method : 0.0888373851776123 length of segment : 399 time for calcul the mask position with numpy : 0.0005068778991699219 nb_pixel_total : 9681 time to create 1 rle with old method : 0.012372255325317383 length of segment : 132 time for calcul the mask position with numpy : 0.0011081695556640625 nb_pixel_total : 31454 time to create 1 rle with old method : 0.03950047492980957 length of segment : 281 time for calcul the mask position with numpy : 0.0006103515625 nb_pixel_total : 18593 time to create 1 rle with old method : 0.022696256637573242 length of segment : 160 time for calcul the mask position with numpy : 0.007714509963989258 nb_pixel_total : 212058 time to create 1 rle with new method : 0.015964508056640625 length of segment : 900 time for calcul the mask position with numpy : 0.002844572067260742 nb_pixel_total : 120712 time to create 1 rle with old method : 0.13702106475830078 length of segment : 409 time for calcul the mask position with numpy : 0.0004761219024658203 nb_pixel_total : 10550 time to create 1 rle with old method : 0.012661218643188477 length of segment : 178 time for calcul the mask position with numpy : 8.654594421386719e-05 nb_pixel_total : 2768 time to create 1 rle with old method : 0.0036656856536865234 length of segment : 46 time for calcul the mask position with numpy : 0.0015208721160888672 nb_pixel_total : 66000 time to create 1 rle with old method : 0.07633280754089355 length of segment : 285 time for calcul the mask position with numpy : 0.000682830810546875 nb_pixel_total : 13815 time to create 1 rle with old method : 0.016556262969970703 length of segment : 224 time for calcul the mask position with numpy : 0.0006465911865234375 nb_pixel_total : 22085 time to create 1 rle with old method : 0.026043415069580078 length of segment : 144 time for calcul the mask position with numpy : 0.0027573108673095703 nb_pixel_total : 159959 time to create 1 rle with new method : 0.0076732635498046875 length of segment : 479 time for calcul the mask position with numpy : 0.0007500648498535156 nb_pixel_total : 37233 time to create 1 rle with old method : 0.04675602912902832 length of segment : 343 time for calcul the mask position with numpy : 0.010219812393188477 nb_pixel_total : 332278 time to create 1 rle with new method : 0.025089740753173828 length of segment : 1217 time for calcul the mask position with numpy : 0.0007276535034179688 nb_pixel_total : 26825 time to create 1 rle with old method : 0.03329873085021973 length of segment : 254 time for calcul the mask position with numpy : 0.2996973991394043 nb_pixel_total : 5126638 time to create 1 rle with new method : 1.034278392791748 length of segment : 3165 time for calcul the mask position with numpy : 0.0005943775177001953 nb_pixel_total : 35204 time to create 1 rle with old method : 0.042724609375 length of segment : 300 time for calcul the mask position with numpy : 0.00021409988403320312 nb_pixel_total : 9929 time to create 1 rle with old method : 0.01349329948425293 length of segment : 51 time for calcul the mask position with numpy : 0.0011997222900390625 nb_pixel_total : 85659 time to create 1 rle with old method : 0.10299944877624512 length of segment : 407 time for calcul the mask position with numpy : 0.0017082691192626953 nb_pixel_total : 56976 time to create 1 rle with old method : 0.06836915016174316 length of segment : 426 time for calcul the mask position with numpy : 0.7818844318389893 nb_pixel_total : 3112069 time to create 1 rle with new method : 1.386888027191162 length of segment : 2705 time for calcul the mask position with numpy : 0.0014100074768066406 nb_pixel_total : 102183 time to create 1 rle with old method : 0.11683845520019531 length of segment : 355 time for calcul the mask position with numpy : 0.012591838836669922 nb_pixel_total : 690563 time to create 1 rle with new method : 0.4864230155944824 length of segment : 1215 time for calcul the mask position with numpy : 0.00022530555725097656 nb_pixel_total : 4075 time to create 1 rle with old method : 0.005122661590576172 length of segment : 72 time for calcul the mask position with numpy : 0.00019216537475585938 nb_pixel_total : 6708 time to create 1 rle with old method : 0.008442401885986328 length of segment : 165 time for calcul the mask position with numpy : 0.0025806427001953125 nb_pixel_total : 130551 time to create 1 rle with old method : 0.14951848983764648 length of segment : 655 time for calcul the mask position with numpy : 0.003844022750854492 nb_pixel_total : 227144 time to create 1 rle with new method : 0.010970354080200195 length of segment : 778 time for calcul the mask position with numpy : 0.0009298324584960938 nb_pixel_total : 27961 time to create 1 rle with old method : 0.03560638427734375 length of segment : 229 time for calcul the mask position with numpy : 0.00029015541076660156 nb_pixel_total : 5038 time to create 1 rle with old method : 0.006504535675048828 length of segment : 99 time for calcul the mask position with numpy : 0.0012066364288330078 nb_pixel_total : 28160 time to create 1 rle with old method : 0.03559279441833496 length of segment : 257 time for calcul the mask position with numpy : 0.0013356208801269531 nb_pixel_total : 51213 time to create 1 rle with old method : 0.06019449234008789 length of segment : 258 time for calcul the mask position with numpy : 0.00019693374633789062 nb_pixel_total : 3918 time to create 1 rle with old method : 0.005087375640869141 length of segment : 50 time for calcul the mask position with numpy : 0.0002014636993408203 nb_pixel_total : 3701 time to create 1 rle with old method : 0.00464940071105957 length of segment : 77 time for calcul the mask position with numpy : 0.007245540618896484 nb_pixel_total : 351492 time to create 1 rle with new method : 0.011572122573852539 length of segment : 676 time for calcul the mask position with numpy : 0.003857851028442383 nb_pixel_total : 122186 time to create 1 rle with old method : 0.13887476921081543 length of segment : 504 time for calcul the mask position with numpy : 0.003627300262451172 nb_pixel_total : 149185 time to create 1 rle with old method : 0.17122721672058105 length of segment : 396 time for calcul the mask position with numpy : 0.006292581558227539 nb_pixel_total : 269975 time to create 1 rle with new method : 0.010215044021606445 length of segment : 590 time for calcul the mask position with numpy : 0.0016705989837646484 nb_pixel_total : 61837 time to create 1 rle with old method : 0.07686328887939453 length of segment : 358 time for calcul the mask position with numpy : 0.0037958621978759766 nb_pixel_total : 64001 time to create 1 rle with old method : 0.07234692573547363 length of segment : 473 time for calcul the mask position with numpy : 0.002724170684814453 nb_pixel_total : 100260 time to create 1 rle with old method : 0.1386861801147461 length of segment : 671 time for calcul the mask position with numpy : 9.608268737792969e-05 nb_pixel_total : 1586 time to create 1 rle with old method : 0.0021581649780273438 length of segment : 37 time for calcul the mask position with numpy : 0.003349781036376953 nb_pixel_total : 73090 time to create 1 rle with old method : 0.08942961692810059 length of segment : 415 time for calcul the mask position with numpy : 0.0021812915802001953 nb_pixel_total : 95520 time to create 1 rle with old method : 0.11396121978759766 length of segment : 252 time for calcul the mask position with numpy : 0.00021791458129882812 nb_pixel_total : 4254 time to create 1 rle with old method : 0.005666017532348633 length of segment : 50 time for calcul the mask position with numpy : 0.0019001960754394531 nb_pixel_total : 50733 time to create 1 rle with old method : 0.06151580810546875 length of segment : 288 time for calcul the mask position with numpy : 0.0012257099151611328 nb_pixel_total : 53884 time to create 1 rle with old method : 0.0665438175201416 length of segment : 421 time for calcul the mask position with numpy : 0.0002751350402832031 nb_pixel_total : 6894 time to create 1 rle with old method : 0.0090179443359375 length of segment : 135 time for calcul the mask position with numpy : 0.0001475811004638672 nb_pixel_total : 1011 time to create 1 rle with old method : 0.0013959407806396484 length of segment : 52 time for calcul the mask position with numpy : 0.004252910614013672 nb_pixel_total : 135960 time to create 1 rle with old method : 0.1508631706237793 length of segment : 537 time for calcul the mask position with numpy : 0.00023937225341796875 nb_pixel_total : 2708 time to create 1 rle with old method : 0.0036318302154541016 length of segment : 117 time for calcul the mask position with numpy : 0.0010056495666503906 nb_pixel_total : 18478 time to create 1 rle with old method : 0.022964954376220703 length of segment : 336 time for calcul the mask position with numpy : 0.00033736228942871094 nb_pixel_total : 5759 time to create 1 rle with old method : 0.007341623306274414 length of segment : 160 time for calcul the mask position with numpy : 0.0005881786346435547 nb_pixel_total : 17002 time to create 1 rle with old method : 0.021626710891723633 length of segment : 199 time for calcul the mask position with numpy : 0.00039577484130859375 nb_pixel_total : 6092 time to create 1 rle with old method : 0.0076143741607666016 length of segment : 179 time for calcul the mask position with numpy : 0.0050811767578125 nb_pixel_total : 126829 time to create 1 rle with old method : 0.14780712127685547 length of segment : 632 time for calcul the mask position with numpy : 0.001725912094116211 nb_pixel_total : 56229 time to create 1 rle with old method : 0.06595826148986816 length of segment : 269 time for calcul the mask position with numpy : 0.001519918441772461 nb_pixel_total : 49504 time to create 1 rle with old method : 0.056481361389160156 length of segment : 325 time for calcul the mask position with numpy : 0.0015523433685302734 nb_pixel_total : 67773 time to create 1 rle with old method : 0.07747817039489746 length of segment : 314 time for calcul the mask position with numpy : 0.001116037368774414 nb_pixel_total : 51481 time to create 1 rle with old method : 0.06371426582336426 length of segment : 264 time for calcul the mask position with numpy : 0.0029447078704833984 nb_pixel_total : 115171 time to create 1 rle with old method : 0.13938021659851074 length of segment : 418 time for calcul the mask position with numpy : 0.0008718967437744141 nb_pixel_total : 20213 time to create 1 rle with old method : 0.02609992027282715 length of segment : 239 time for calcul the mask position with numpy : 0.0005080699920654297 nb_pixel_total : 18774 time to create 1 rle with old method : 0.02391362190246582 length of segment : 218 time for calcul the mask position with numpy : 0.0001418590545654297 nb_pixel_total : 2502 time to create 1 rle with old method : 0.003423929214477539 length of segment : 67 time for calcul the mask position with numpy : 0.0004899501800537109 nb_pixel_total : 8185 time to create 1 rle with old method : 0.011176824569702148 length of segment : 109 time for calcul the mask position with numpy : 0.0009865760803222656 nb_pixel_total : 28943 time to create 1 rle with old method : 0.03706669807434082 length of segment : 228 time for calcul the mask position with numpy : 0.001580953598022461 nb_pixel_total : 114572 time to create 1 rle with old method : 0.14229440689086914 length of segment : 411 time for calcul the mask position with numpy : 0.00020694732666015625 nb_pixel_total : 2785 time to create 1 rle with old method : 0.0038716793060302734 length of segment : 48 time for calcul the mask position with numpy : 0.0012996196746826172 nb_pixel_total : 40773 time to create 1 rle with old method : 0.04952049255371094 length of segment : 491 time for calcul the mask position with numpy : 0.0035762786865234375 nb_pixel_total : 162107 time to create 1 rle with new method : 0.005435943603515625 length of segment : 563 time for calcul the mask position with numpy : 0.0003829002380371094 nb_pixel_total : 9290 time to create 1 rle with old method : 0.012186050415039062 length of segment : 122 time for calcul the mask position with numpy : 0.0001723766326904297 nb_pixel_total : 1681 time to create 1 rle with old method : 0.002422332763671875 length of segment : 53 time for calcul the mask position with numpy : 0.0017549991607666016 nb_pixel_total : 31746 time to create 1 rle with old method : 0.0381162166595459 length of segment : 210 time for calcul the mask position with numpy : 0.00035834312438964844 nb_pixel_total : 8300 time to create 1 rle with old method : 0.010837078094482422 length of segment : 128 time for calcul the mask position with numpy : 0.000286102294921875 nb_pixel_total : 1364 time to create 1 rle with old method : 0.002801179885864258 length of segment : 67 time for calcul the mask position with numpy : 0.005288839340209961 nb_pixel_total : 188849 time to create 1 rle with new method : 0.01086735725402832 length of segment : 524 time spent for convertir_results : 25.101183652877808 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 138 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Number RLEs to save : 53675 save missing photos in datou_result : time spend for datou_step_exec : 71.92762541770935 time spend to save output : 4.261010408401489 total time spend for step 1 : 76.18863582611084 step2:crop_condition Sat Mar 8 00:19:46 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 138 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 10 About to insert : list_path_to_insert length 10 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 35 About to insert : list_path_to_insert length 35 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 14 About to insert : list_path_to_insert length 14 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1341321334, 1341321328, 1341321324, 1341266663, 1341266213, 1341266207, 1341266194, 1341266186, 1341265999, 1341265987] Looping around the photos to save general results len do output : 78 /-3698330427Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330423Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330430Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330446Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598122Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598147Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598169Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330435Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598093Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330471Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598111Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598135Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330479Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330476Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330426Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330439Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330455Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330460Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330469Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598106Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598134Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598139Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598156Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598171Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330429Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330428Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330437Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330434Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330424Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330436Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330451Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330447Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330450Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330444Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330449Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330465Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330464Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598096Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598095Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598110Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598097Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330467Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598101Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598105Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330473Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330475Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598142Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330478Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330477Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598132Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598153Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598155Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598161Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598152Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598162Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598151Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598154Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598165Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330480Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330448Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330445Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330441Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330461Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598100Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598103Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598098Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598104Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330472Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598118Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598129Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598158Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598168Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598166Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330442Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598131Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330438Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3698330452Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3696598157Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341321334', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341321328', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341321324', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266663', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266213', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266207', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266194', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266186', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341265999', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341265987', None, None, None, None, None, '2650228') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 244 time used for this insertion : 0.052942514419555664 save_final save missing photos in datou_result : time spend for datou_step_exec : 42.99736452102661 time spend to save output : 0.055388689041137695 total time spend for step 2 : 43.05275321006775 step3:thcl Sat Mar 8 00:20:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.011672735214233398 time to convert the images to numpy array : 0.07431411743164062 time to import caffe and check if the image exist : 0.01612091064453125 time to convert the images to numpy array : 0.34476518630981445 time to import caffe and check if the image exist : 0.014816999435424805 time to convert the images to numpy array : 0.35311031341552734 time to import caffe and check if the image exist : 0.015577316284179688 time to convert the images to numpy array : 0.37767863273620605 time to import caffe and check if the image exist : 0.012633562088012695 time to convert the images to numpy array : 0.4644169807434082 time to import caffe and check if the image exist : 0.018848896026611328 time to convert the images to numpy array : 0.4647543430328369 time to import caffe and check if the image exist : 0.01496577262878418 time to convert the images to numpy array : 0.6364986896514893 time to import caffe and check if the image exist : 0.020990610122680664 time to convert the images to numpy array : 0.6592109203338623 time to import caffe and check if the image exist : 0.010630369186401367 time to convert the images to numpy array : 0.6954629421234131 time to import caffe and check if the image exist : 0.02091670036315918 time to convert the images to numpy array : 0.7373933792114258 total time to convert the images to numpy array : 1.1343636512756348 list photo_ids error: [] list photo_ids correct : [-3696598103, -3696598098, -3696598104, -3698330472, -3696598118, -3696598129, -3696598158, -3696598168, -3696598154, -3696598165, -3698330480, -3698330448, -3698330445, -3698330441, -3698330461, -3696598100, -3696598166, -3698330442, -3696598131, -3698330438, -3698330452, -3696598157, -3698330429, -3698330428, -3698330437, -3698330434, -3698330424, -3698330436, -3698330451, -3698330447, -3698330455, -3698330460, -3698330469, -3696598106, -3696598134, -3696598139, -3696598156, -3696598171, -3696598093, -3698330471, -3696598111, -3696598135, -3698330479, -3698330476, -3698330426, -3698330439, -3698330477, -3696598132, -3696598153, -3696598155, -3696598161, -3696598152, -3696598162, -3696598151, -3696598097, -3698330467, -3696598101, -3696598105, -3698330473, -3698330475, -3696598142, -3698330478, -3698330427, -3698330423, -3698330430, -3698330446, -3696598122, -3696598147, -3696598169, -3698330435, -3698330450, -3698330444, -3698330449, -3698330465, -3698330464, -3696598096, -3696598095, -3696598110] number of photos to traite : 78 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5178 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 4653 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.3722031116485596 time used to do the prediction : 0.2739715576171875 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3927 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3047 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.3652229309082031 time used to do the prediction : 0.16612648963928223 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 2.2172927856445312e-05 save missing photos in datou_result : time spend for datou_step_exec : 11.834996938705444 time spend to save output : 0.00067138671875 total time spend for step 3 : 11.835668325424194 step4:merge_mask_thcl_custom Sat Mar 8 00:20:41 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 138 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1341321334, 1341321328, 1341321324, 1341266663, 1341266213, 1341266207, 1341266194, 1341266186, 1341265999, 1341265987] Looping around the photos to save general results len do output : 10 /1341321334Didn't retrieve data .Didn't retrieve data . /1341321328Didn't retrieve data .Didn't retrieve data . /1341321324Didn't retrieve data .Didn't retrieve data . /1341266663Didn't retrieve data .Didn't retrieve data . /1341266213Didn't retrieve data .Didn't retrieve data . /1341266207Didn't retrieve data .Didn't retrieve data . /1341266194Didn't retrieve data .Didn't retrieve data . /1341266186Didn't retrieve data .Didn't retrieve data . /1341265999Didn't retrieve data .Didn't retrieve data . /1341265987Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341321334', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341321328', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341321324', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266663', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266213', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266207', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266194', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266186', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341265999', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341265987', None, None, None, None, None, '2650228') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.03250408172607422 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.07402849197387695 time spend to save output : 0.03331708908081055 total time spend for step 4 : 0.1073455810546875 step5:rle_unique_nms_with_priority Sat Mar 8 00:20:41 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 17 nb_hashtags : 7 time to prepare the origin masks : 14.029016494750977 time for calcul the mask position with numpy : 1.0911521911621094 nb_pixel_total : 5893064 time to create 1 rle with new method : 0.6864299774169922 time for calcul the mask position with numpy : 0.03491473197937012 nb_pixel_total : 16591 time to create 1 rle with old method : 0.020591020584106445 time for calcul the mask position with numpy : 0.03686404228210449 nb_pixel_total : 23691 time to create 1 rle with old method : 0.028475522994995117 time for calcul the mask position with numpy : 0.03648805618286133 nb_pixel_total : 50299 time to create 1 rle with old method : 0.058612823486328125 time for calcul the mask position with numpy : 0.04290175437927246 nb_pixel_total : 603099 time to create 1 rle with new method : 0.9069573879241943 time for calcul the mask position with numpy : 0.03948831558227539 nb_pixel_total : 417 time to create 1 rle with old method : 0.0012087821960449219 time for calcul the mask position with numpy : 0.03750300407409668 nb_pixel_total : 88952 time to create 1 rle with old method : 0.0950307846069336 time for calcul the mask position with numpy : 0.03625988960266113 nb_pixel_total : 50505 time to create 1 rle with old method : 0.08144640922546387 time for calcul the mask position with numpy : 0.03806352615356445 nb_pixel_total : 2456 time to create 1 rle with old method : 0.003172159194946289 time for calcul the mask position with numpy : 0.042352914810180664 nb_pixel_total : 2289 time to create 1 rle with old method : 0.0030279159545898438 time for calcul the mask position with numpy : 0.03766298294067383 nb_pixel_total : 61791 time to create 1 rle with old method : 0.08108067512512207 time for calcul the mask position with numpy : 0.036873579025268555 nb_pixel_total : 8809 time to create 1 rle with old method : 0.013030290603637695 time for calcul the mask position with numpy : 0.034004926681518555 nb_pixel_total : 28549 time to create 1 rle with old method : 0.03292441368103027 time for calcul the mask position with numpy : 0.03435039520263672 nb_pixel_total : 105692 time to create 1 rle with old method : 0.12322497367858887 time for calcul the mask position with numpy : 0.04042863845825195 nb_pixel_total : 75002 time to create 1 rle with old method : 0.08551549911499023 time for calcul the mask position with numpy : 0.0376744270324707 nb_pixel_total : 16194 time to create 1 rle with old method : 0.018384933471679688 time for calcul the mask position with numpy : 0.026025056838989258 nb_pixel_total : 8415 time to create 1 rle with old method : 0.009257316589355469 time for calcul the mask position with numpy : 0.023053407669067383 nb_pixel_total : 14425 time to create 1 rle with old method : 0.017434358596801758 create new chi : 4.0330705642700195 time to delete rle : 0.019913673400878906 batch 1 Loaded 18 chid ids of type : 3726 Number RLEs to save : 12143 TO DO : save crop sub photo not yet done ! save time : 0.9237263202667236 nb_obj : 13 nb_hashtags : 7 time to prepare the origin masks : 14.982372760772705 time for calcul the mask position with numpy : 0.31450653076171875 nb_pixel_total : 6629304 time to create 1 rle with new method : 0.5415439605712891 time for calcul the mask position with numpy : 0.029604673385620117 nb_pixel_total : 41727 time to create 1 rle with old method : 0.0464932918548584 time for calcul the mask position with numpy : 0.03172183036804199 nb_pixel_total : 49383 time to create 1 rle with old method : 0.05804920196533203 time for calcul the mask position with numpy : 0.028747081756591797 nb_pixel_total : 5454 time to create 1 rle with old method : 0.009202003479003906 time for calcul the mask position with numpy : 0.02775263786315918 nb_pixel_total : 25393 time to create 1 rle with old method : 0.029603242874145508 time for calcul the mask position with numpy : 0.026563405990600586 nb_pixel_total : 33588 time to create 1 rle with old method : 0.03823709487915039 time for calcul the mask position with numpy : 0.025249004364013672 nb_pixel_total : 17493 time to create 1 rle with old method : 0.019708871841430664 time for calcul the mask position with numpy : 0.025826454162597656 nb_pixel_total : 3624 time to create 1 rle with old method : 0.004801750183105469 time for calcul the mask position with numpy : 0.026889801025390625 nb_pixel_total : 23960 time to create 1 rle with old method : 0.026903390884399414 time for calcul the mask position with numpy : 0.030143260955810547 nb_pixel_total : 168745 time to create 1 rle with new method : 0.6425533294677734 time for calcul the mask position with numpy : 0.027036190032958984 nb_pixel_total : 1418 time to create 1 rle with old method : 0.001982450485229492 time for calcul the mask position with numpy : 0.032755374908447266 nb_pixel_total : 33399 time to create 1 rle with old method : 0.03947806358337402 time for calcul the mask position with numpy : 0.027354955673217773 nb_pixel_total : 11574 time to create 1 rle with old method : 0.013704776763916016 time for calcul the mask position with numpy : 0.026316165924072266 nb_pixel_total : 5178 time to create 1 rle with old method : 0.0059778690338134766 create new chi : 2.2337679862976074 time to delete rle : 0.0010020732879638672 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 7594 TO DO : save crop sub photo not yet done ! save time : 0.7968332767486572 nb_obj : 10 nb_hashtags : 3 time to prepare the origin masks : 8.801719665527344 time for calcul the mask position with numpy : 0.06760668754577637 nb_pixel_total : 2461621 time to create 1 rle with new method : 3.003662109375 time for calcul the mask position with numpy : 0.03840804100036621 nb_pixel_total : 127823 time to create 1 rle with old method : 0.15014410018920898 time for calcul the mask position with numpy : 0.038701772689819336 nb_pixel_total : 4112 time to create 1 rle with old method : 0.00538945198059082 time for calcul the mask position with numpy : 0.0355534553527832 nb_pixel_total : 47261 time to create 1 rle with old method : 0.05753445625305176 time for calcul the mask position with numpy : 0.033254146575927734 nb_pixel_total : 1155 time to create 1 rle with old method : 0.001355886459350586 time for calcul the mask position with numpy : 0.036019086837768555 nb_pixel_total : 2393 time to create 1 rle with old method : 0.002790212631225586 time for calcul the mask position with numpy : 0.0360260009765625 nb_pixel_total : 438078 time to create 1 rle with new method : 0.4821484088897705 time for calcul the mask position with numpy : 0.2902660369873047 nb_pixel_total : 3907601 time to create 1 rle with new method : 1.2989530563354492 time for calcul the mask position with numpy : 0.02261495590209961 nb_pixel_total : 28005 time to create 1 rle with old method : 0.04514622688293457 time for calcul the mask position with numpy : 0.02350449562072754 nb_pixel_total : 15152 time to create 1 rle with old method : 0.019097566604614258 time for calcul the mask position with numpy : 0.02216362953186035 nb_pixel_total : 17039 time to create 1 rle with old method : 0.029775619506835938 create new chi : 5.832757234573364 time to delete rle : 0.0011167526245117188 batch 1 Loaded 11 chid ids of type : 3726 Number RLEs to save : 11563 TO DO : save crop sub photo not yet done ! save time : 0.8081936836242676 nb_obj : 5 nb_hashtags : 3 time to prepare the origin masks : 12.27745246887207 time for calcul the mask position with numpy : 0.03594684600830078 nb_pixel_total : 1397598 time to create 1 rle with new method : 0.9012057781219482 time for calcul the mask position with numpy : 0.022714853286743164 nb_pixel_total : 46058 time to create 1 rle with old method : 0.060675859451293945 time for calcul the mask position with numpy : 0.02310943603515625 nb_pixel_total : 68588 time to create 1 rle with old method : 0.07976603507995605 time for calcul the mask position with numpy : 0.021582841873168945 nb_pixel_total : 32938 time to create 1 rle with old method : 0.03888130187988281 time for calcul the mask position with numpy : 0.02196979522705078 nb_pixel_total : 17893 time to create 1 rle with old method : 0.023788928985595703 time for calcul the mask position with numpy : 0.4295217990875244 nb_pixel_total : 5487165 time to create 1 rle with new method : 0.8820037841796875 create new chi : 2.610203742980957 time to delete rle : 0.0013272762298583984 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 14115 TO DO : save crop sub photo not yet done ! save time : 1.2363872528076172 nb_obj : 14 nb_hashtags : 5 time to prepare the origin masks : 14.054493188858032 time for calcul the mask position with numpy : 0.5629792213439941 nb_pixel_total : 6113607 time to create 1 rle with new method : 1.3087272644042969 time for calcul the mask position with numpy : 0.026837587356567383 nb_pixel_total : 62975 time to create 1 rle with old method : 0.09141850471496582 time for calcul the mask position with numpy : 0.028380632400512695 nb_pixel_total : 72951 time to create 1 rle with old method : 0.0811915397644043 time for calcul the mask position with numpy : 0.026772260665893555 nb_pixel_total : 65820 time to create 1 rle with old method : 0.08391714096069336 time for calcul the mask position with numpy : 0.029927730560302734 nb_pixel_total : 18538 time to create 1 rle with old method : 0.026574373245239258 time for calcul the mask position with numpy : 0.02980494499206543 nb_pixel_total : 2752 time to create 1 rle with old method : 0.0031414031982421875 time for calcul the mask position with numpy : 0.02376842498779297 nb_pixel_total : 515 time to create 1 rle with old method : 0.0010542869567871094 time for calcul the mask position with numpy : 0.024785280227661133 nb_pixel_total : 160340 time to create 1 rle with new method : 0.6883213520050049 time for calcul the mask position with numpy : 0.024502992630004883 nb_pixel_total : 9641 time to create 1 rle with old method : 0.010451793670654297 time for calcul the mask position with numpy : 0.022194862365722656 nb_pixel_total : 13777 time to create 1 rle with old method : 0.014370203018188477 time for calcul the mask position with numpy : 0.023075342178344727 nb_pixel_total : 10511 time to create 1 rle with old method : 0.011273860931396484 time for calcul the mask position with numpy : 0.030684709548950195 nb_pixel_total : 330206 time to create 1 rle with new method : 1.0935955047607422 time for calcul the mask position with numpy : 0.024082183837890625 nb_pixel_total : 31400 time to create 1 rle with old method : 0.032961130142211914 time for calcul the mask position with numpy : 0.023116350173950195 nb_pixel_total : 120595 time to create 1 rle with old method : 0.12566375732421875 time for calcul the mask position with numpy : 0.024519681930541992 nb_pixel_total : 36612 time to create 1 rle with old method : 0.0383298397064209 create new chi : 4.630087852478027 time to delete rle : 0.002193450927734375 batch 1 Loaded 17 chid ids of type : 3726 Number RLEs to save : 12227 TO DO : save crop sub photo not yet done ! save time : 4.480304956436157 nb_obj : 6 nb_hashtags : 4 time to prepare the origin masks : 7.340453624725342 time for calcul the mask position with numpy : 0.05668783187866211 nb_pixel_total : 1714135 time to create 1 rle with new method : 0.7631001472473145 time for calcul the mask position with numpy : 0.03521609306335449 nb_pixel_total : 7449 time to create 1 rle with old method : 0.008455038070678711 time for calcul the mask position with numpy : 0.03352999687194824 nb_pixel_total : 14959 time to create 1 rle with old method : 0.016708850860595703 time for calcul the mask position with numpy : 0.022197484970092773 nb_pixel_total : 106140 time to create 1 rle with old method : 0.12968826293945312 time for calcul the mask position with numpy : 0.024074077606201172 nb_pixel_total : 18641 time to create 1 rle with old method : 0.02219557762145996 time for calcul the mask position with numpy : 0.3565647602081299 nb_pixel_total : 5103439 time to create 1 rle with new method : 1.2637815475463867 time for calcul the mask position with numpy : 0.03178572654724121 nb_pixel_total : 85477 time to create 1 rle with old method : 0.09393477439880371 create new chi : 2.9203028678894043 time to delete rle : 0.001523733139038086 batch 1 Loaded 12 chid ids of type : 3726 Number RLEs to save : 11431 TO DO : save crop sub photo not yet done ! save time : 0.8315775394439697 nb_obj : 9 nb_hashtags : 4 time to prepare the origin masks : 10.539202213287354 time for calcul the mask position with numpy : 0.9675540924072266 nb_pixel_total : 3389134 time to create 1 rle with new method : 1.6216444969177246 time for calcul the mask position with numpy : 0.036949872970581055 nb_pixel_total : 102055 time to create 1 rle with old method : 0.1207423210144043 time for calcul the mask position with numpy : 0.03804731369018555 nb_pixel_total : 108956 time to create 1 rle with old method : 0.12574291229248047 time for calcul the mask position with numpy : 0.03498482704162598 nb_pixel_total : 101970 time to create 1 rle with old method : 0.12607336044311523 time for calcul the mask position with numpy : 0.03529977798461914 nb_pixel_total : 5480 time to create 1 rle with old method : 0.006425142288208008 time for calcul the mask position with numpy : 0.03024435043334961 nb_pixel_total : 4062 time to create 1 rle with old method : 0.004741191864013672 time for calcul the mask position with numpy : 0.023013591766357422 nb_pixel_total : 152833 time to create 1 rle with new method : 1.0015408992767334 time for calcul the mask position with numpy : 0.036435604095458984 nb_pixel_total : 6999 time to create 1 rle with old method : 0.010579824447631836 time for calcul the mask position with numpy : 0.5650129318237305 nb_pixel_total : 3121989 time to create 1 rle with new method : 1.518306016921997 time for calcul the mask position with numpy : 0.03632807731628418 nb_pixel_total : 56762 time to create 1 rle with old method : 0.06564617156982422 create new chi : 6.503529787063599 time to delete rle : 0.001870870590209961 batch 1 Loaded 11 chid ids of type : 3726 Number RLEs to save : 14412 TO DO : save crop sub photo not yet done ! save time : 1.2521085739135742 nb_obj : 21 nb_hashtags : 7 time to prepare the origin masks : 25.44345211982727 time for calcul the mask position with numpy : 0.39901089668273926 nb_pixel_total : 5492003 time to create 1 rle with new method : 1.1843810081481934 time for calcul the mask position with numpy : 0.024382591247558594 nb_pixel_total : 121987 time to create 1 rle with old method : 0.14007782936096191 time for calcul the mask position with numpy : 0.026862382888793945 nb_pixel_total : 61850 time to create 1 rle with old method : 0.07296276092529297 time for calcul the mask position with numpy : 0.025942564010620117 nb_pixel_total : 148661 time to create 1 rle with old method : 0.16480302810668945 time for calcul the mask position with numpy : 0.02658677101135254 nb_pixel_total : 72910 time to create 1 rle with old method : 0.0909111499786377 time for calcul the mask position with numpy : 0.02427840232849121 nb_pixel_total : 6850 time to create 1 rle with old method : 0.047853708267211914 time for calcul the mask position with numpy : 0.04340338706970215 nb_pixel_total : 1012 time to create 1 rle with old method : 0.0011811256408691406 time for calcul the mask position with numpy : 0.04020237922668457 nb_pixel_total : 3683 time to create 1 rle with old method : 0.004314899444580078 time for calcul the mask position with numpy : 0.04012489318847656 nb_pixel_total : 4220 time to create 1 rle with old method : 0.004857063293457031 time for calcul the mask position with numpy : 0.047490835189819336 nb_pixel_total : 100212 time to create 1 rle with old method : 0.11541748046875 time for calcul the mask position with numpy : 0.04230523109436035 nb_pixel_total : 1575 time to create 1 rle with old method : 0.0017247200012207031 time for calcul the mask position with numpy : 0.041029930114746094 nb_pixel_total : 3665 time to create 1 rle with old method : 0.004340410232543945 time for calcul the mask position with numpy : 0.040969133377075195 nb_pixel_total : 51154 time to create 1 rle with old method : 0.054903507232666016 time for calcul the mask position with numpy : 0.0468599796295166 nb_pixel_total : 63790 time to create 1 rle with old method : 0.07398223876953125 time for calcul the mask position with numpy : 0.04518914222717285 nb_pixel_total : 269250 time to create 1 rle with new method : 1.1112473011016846 time for calcul the mask position with numpy : 0.0493474006652832 nb_pixel_total : 351413 time to create 1 rle with new method : 0.7850503921508789 time for calcul the mask position with numpy : 0.04327845573425293 nb_pixel_total : 3890 time to create 1 rle with old method : 0.007387876510620117 time for calcul the mask position with numpy : 0.04262971878051758 nb_pixel_total : 135961 time to create 1 rle with old method : 0.20086002349853516 time for calcul the mask position with numpy : 0.036484479904174805 nb_pixel_total : 28017 time to create 1 rle with old method : 0.03218579292297363 time for calcul the mask position with numpy : 0.03314399719238281 nb_pixel_total : 4998 time to create 1 rle with old method : 0.005661964416503906 time for calcul the mask position with numpy : 0.03243875503540039 nb_pixel_total : 27881 time to create 1 rle with old method : 0.03092646598815918 time for calcul the mask position with numpy : 0.029445648193359375 nb_pixel_total : 95258 time to create 1 rle with old method : 0.1161658763885498 create new chi : 5.5227789878845215 time to delete rle : 0.0020928382873535156 batch 1 Loaded 26 chid ids of type : 3726 Number RLEs to save : 14690 TO DO : save crop sub photo not yet done ! save time : 1.3565583229064941 nb_obj : 18 nb_hashtags : 6 time to prepare the origin masks : 18.68099021911621 time for calcul the mask position with numpy : 0.6760389804840088 nb_pixel_total : 6469250 time to create 1 rle with new method : 1.212852954864502 time for calcul the mask position with numpy : 0.03922390937805176 nb_pixel_total : 28833 time to create 1 rle with old method : 0.03303813934326172 time for calcul the mask position with numpy : 0.03604769706726074 nb_pixel_total : 67685 time to create 1 rle with old method : 0.07812809944152832 time for calcul the mask position with numpy : 0.03750944137573242 nb_pixel_total : 1803 time to create 1 rle with old method : 0.002610921859741211 time for calcul the mask position with numpy : 0.03608822822570801 nb_pixel_total : 3454 time to create 1 rle with old method : 0.004496574401855469 time for calcul the mask position with numpy : 0.036951303482055664 nb_pixel_total : 2753 time to create 1 rle with old method : 0.003136873245239258 time for calcul the mask position with numpy : 0.03702831268310547 nb_pixel_total : 126663 time to create 1 rle with old method : 0.14525580406188965 time for calcul the mask position with numpy : 0.037778377532958984 nb_pixel_total : 18765 time to create 1 rle with old method : 0.021996498107910156 time for calcul the mask position with numpy : 0.03796529769897461 nb_pixel_total : 8069 time to create 1 rle with old method : 0.01013803482055664 time for calcul the mask position with numpy : 0.03901863098144531 nb_pixel_total : 2486 time to create 1 rle with old method : 0.0035881996154785156 time for calcul the mask position with numpy : 0.04369640350341797 nb_pixel_total : 55988 time to create 1 rle with old method : 0.0646672248840332 time for calcul the mask position with numpy : 0.03831934928894043 nb_pixel_total : 113851 time to create 1 rle with old method : 0.13251733779907227 time for calcul the mask position with numpy : 0.03583264350891113 nb_pixel_total : 49415 time to create 1 rle with old method : 0.056914567947387695 time for calcul the mask position with numpy : 0.037108421325683594 nb_pixel_total : 51363 time to create 1 rle with old method : 0.05887436866760254 time for calcul the mask position with numpy : 0.03565526008605957 nb_pixel_total : 18400 time to create 1 rle with old method : 0.02261829376220703 time for calcul the mask position with numpy : 0.03692173957824707 nb_pixel_total : 6084 time to create 1 rle with old method : 0.006861209869384766 time for calcul the mask position with numpy : 0.03543829917907715 nb_pixel_total : 16926 time to create 1 rle with old method : 0.02049708366394043 time for calcul the mask position with numpy : 0.04525899887084961 nb_pixel_total : 5750 time to create 1 rle with old method : 0.006772518157958984 time for calcul the mask position with numpy : 0.039873361587524414 nb_pixel_total : 2702 time to create 1 rle with old method : 0.0031952857971191406 create new chi : 3.2933778762817383 time to delete rle : 0.0011572837829589844 batch 1 Loaded 19 chid ids of type : 3726 Number RLEs to save : 10510 TO DO : save crop sub photo not yet done ! save time : 0.9322283267974854 nb_obj : 7 nb_hashtags : 4 time to prepare the origin masks : 10.114917516708374 time for calcul the mask position with numpy : 0.828751802444458 nb_pixel_total : 6608471 time to create 1 rle with new method : 1.047581434249878 time for calcul the mask position with numpy : 0.04172921180725098 nb_pixel_total : 9248 time to create 1 rle with old method : 0.010262489318847656 time for calcul the mask position with numpy : 0.042806148529052734 nb_pixel_total : 1662 time to create 1 rle with old method : 0.001956462860107422 time for calcul the mask position with numpy : 0.0452885627746582 nb_pixel_total : 188281 time to create 1 rle with new method : 0.7073996067047119 time for calcul the mask position with numpy : 0.04264640808105469 nb_pixel_total : 8287 time to create 1 rle with old method : 0.009671688079833984 time for calcul the mask position with numpy : 0.027201414108276367 nb_pixel_total : 31565 time to create 1 rle with old method : 0.03955578804016113 time for calcul the mask position with numpy : 0.03287482261657715 nb_pixel_total : 161942 time to create 1 rle with new method : 1.3963794708251953 time for calcul the mask position with numpy : 0.026921987533569336 nb_pixel_total : 40784 time to create 1 rle with old method : 0.05157899856567383 create new chi : 4.458524465560913 time to delete rle : 0.001012563705444336 batch 1 Loaded 9 chid ids of type : 3726 Number RLEs to save : 6360 TO DO : save crop sub photo not yet done ! save time : 0.49862098693847656 map_output_result : {1341321334: (0.0, 'Should be the crop_list due to order', 0), 1341321328: (0.0, 'Should be the crop_list due to order', 0), 1341321324: (0.0, 'Should be the crop_list due to order', 0), 1341266663: (0.0, 'Should be the crop_list due to order', 0), 1341266213: (0.0, 'Should be the crop_list due to order', 0), 1341266207: (0.0, 'Should be the crop_list due to order', 0), 1341266194: (0.0, 'Should be the crop_list due to order', 0), 1341266186: (0.0, 'Should be the crop_list due to order', 0), 1341265999: (0.0, 'Should be the crop_list due to order', 0), 1341265987: (0.0, 'Should be the crop_list due to order', 0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1341321334, 1341321328, 1341321324, 1341266663, 1341266213, 1341266207, 1341266194, 1341266186, 1341265999, 1341265987] Looping around the photos to save general results len do output : 10 /1341321334.Didn't retrieve data . /1341321328.Didn't retrieve data . /1341321324.Didn't retrieve data . /1341266663.Didn't retrieve data . /1341266213.Didn't retrieve data . /1341266207.Didn't retrieve data . /1341266194.Didn't retrieve data . /1341266186.Didn't retrieve data . /1341265999.Didn't retrieve data . /1341265987.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341321334', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341321328', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341321324', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266663', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266213', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266207', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266194', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266186', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341265999', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341265987', None, None, None, None, None, '2650228') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.04788708686828613 save_final save missing photos in datou_result : time spend for datou_step_exec : 192.71688389778137 time spend to save output : 0.07230448722839355 total time spend for step 5 : 192.78918838500977 step6:crop_condition Sat Mar 8 00:23:54 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 145 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741389836_1427269 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 4.925200939178467 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! About to upload 7 photos upload in portfolio : 4869462 init cache_photo without model_param we have 7 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741389844_1427269 we have uploaded 7 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.488528251647949 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741389849_1427269 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 3.0790622234344482 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! About to upload 5 photos upload in portfolio : 4869462 init cache_photo without model_param we have 5 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741389854_1427269 we have uploaded 5 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.1931722164154053 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741389858_1427269 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.2295963764190674 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741389861_1427269 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.5833399295806885 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741389866_1427269 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.599161148071289 we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1341321334, 1341321328, 1341321324, 1341266663, 1341266213, 1341266207, 1341266194, 1341266186, 1341265999, 1341265987] Looping around the photos to save general results len do output : 26 /1341636340Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341636343Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341636344Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341636370Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341636371Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341636374Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341636376Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341636377Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341636380Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341636381Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341636427Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341636430Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341636436Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341636462Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341636466Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341636467Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341636469Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341636473Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341636508Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341636509Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341636510Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341636547Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341636548Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341636588Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341636590Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341636592Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341321334', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341321328', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341321324', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266663', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266213', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266207', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266194', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266186', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341265999', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341265987', None, None, None, None, None, '2650228') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 88 time used for this insertion : 0.02933645248413086 save_final save missing photos in datou_result : time spend for datou_step_exec : 33.86479425430298 time spend to save output : 0.03043651580810547 total time spend for step 6 : 33.895230770111084 step7:ventilate_hashtags_in_portfolio Sat Mar 8 00:24:28 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 21164371 get user id for portfolio 21164371 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21164371 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('mal_croppe','cartonnette','autre_refus','papier','Carton_gris','environnement','Carton_brun','Teint_Dans_La_Masse','kraft','metal','plastique','flou')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21164371 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('mal_croppe','cartonnette','autre_refus','papier','Carton_gris','environnement','Carton_brun','Teint_Dans_La_Masse','kraft','metal','plastique','flou')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21164371 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('mal_croppe','cartonnette','autre_refus','papier','Carton_gris','environnement','Carton_brun','Teint_Dans_La_Masse','kraft','metal','plastique','flou')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/21164625,21164626,21164627,21164628,21164629,21164630,21164631,21164632,21164633,21164634,21164635,21164636?tags=mal_croppe,cartonnette,autre_refus,papier,Carton_gris,environnement,Carton_brun,Teint_Dans_La_Masse,kraft,metal,plastique,flou Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1341321334, 1341321328, 1341321324, 1341266663, 1341266213, 1341266207, 1341266194, 1341266186, 1341265999, 1341265987] Looping around the photos to save general results len do output : 1 /21164371. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341321334', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341321328', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341321324', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266663', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266213', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266207', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266194', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266186', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341265999', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341265987', None, None, None, None, None, '2650228') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.04126119613647461 save_final save missing photos in datou_result : time spend for datou_step_exec : 2.1487224102020264 time spend to save output : 0.041553497314453125 total time spend for step 7 : 2.1902759075164795 step8:final Sat Mar 8 00:24:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1341321334: ('0.06356172894348543',), 1341321328: ('0.06356172894348543',), 1341321324: ('0.06356172894348543',), 1341266663: ('0.06356172894348543',), 1341266213: ('0.06356172894348543',), 1341266207: ('0.06356172894348543',), 1341266194: ('0.06356172894348543',), 1341266186: ('0.06356172894348543',), 1341265999: ('0.06356172894348543',), 1341265987: ('0.06356172894348543',)} new output for save of step final : {1341321334: ('0.06356172894348543',), 1341321328: ('0.06356172894348543',), 1341321324: ('0.06356172894348543',), 1341266663: ('0.06356172894348543',), 1341266213: ('0.06356172894348543',), 1341266207: ('0.06356172894348543',), 1341266194: ('0.06356172894348543',), 1341266186: ('0.06356172894348543',), 1341265999: ('0.06356172894348543',), 1341265987: ('0.06356172894348543',)} [1341321334, 1341321328, 1341321324, 1341266663, 1341266213, 1341266207, 1341266194, 1341266186, 1341265999, 1341265987] Looping around the photos to save general results len do output : 10 /1341321334.Didn't retrieve data . /1341321328.Didn't retrieve data . /1341321324.Didn't retrieve data . /1341266663.Didn't retrieve data . /1341266213.Didn't retrieve data . /1341266207.Didn't retrieve data . /1341266194.Didn't retrieve data . /1341266186.Didn't retrieve data . /1341265999.Didn't retrieve data . /1341265987.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341321334', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341321328', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341321324', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266663', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266213', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266207', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266194', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266186', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341265999', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341265987', None, None, None, None, None, '2650228') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.018786907196044922 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.2846238613128662 time spend to save output : 0.019635438919067383 total time spend for step 8 : 0.3042593002319336 step9:velours_tree Sat Mar 8 00:24:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.18096494674682617 time spend to save output : 5.221366882324219e-05 total time spend for step 9 : 0.18101716041564941 step10:send_mail_cod Sat Mar 8 00:24:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P21164371_08-03-2025_00_24_30.pdf 21164625 imagette211646251741389870 21164626 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette211646261741389870 21164627 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette211646271741389871 21164633 change filename to text .change filename to text .change filename to text .imagette211646331741389872 21164634 imagette211646341741389872 21164635 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette211646351741389872 21164636 imagette211646361741389872 21164629 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette211646291741389872 21164631 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette211646311741389873 21164632 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette211646321741389874 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=21164371 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/21164625,21164626,21164627,21164628,21164629,21164630,21164631,21164632,21164633,21164634,21164635,21164636?tags=mal_croppe,cartonnette,autre_refus,papier,Carton_gris,environnement,Carton_brun,Teint_Dans_La_Masse,kraft,metal,plastique,flou your option no_mail is active, we will not send the real mail to your client args[1341321334] : ('0.06356172894348543',) no score found for photo 1341321334 We are sending mail with results at report@fotonower.com args[1341321328] : ('0.06356172894348543',) no score found for photo 1341321328 We are sending mail with results at report@fotonower.com args[1341321324] : ('0.06356172894348543',) no score found for photo 1341321324 We are sending mail with results at report@fotonower.com args[1341266663] : ('0.06356172894348543',) no score found for photo 1341266663 We are sending mail with results at report@fotonower.com args[1341266213] : ('0.06356172894348543',) no score found for photo 1341266213 We are sending mail with results at report@fotonower.com args[1341266207] : ('0.06356172894348543',) no score found for photo 1341266207 We are sending mail with results at report@fotonower.com args[1341266194] : ('0.06356172894348543',) no score found for photo 1341266194 We are sending mail with results at report@fotonower.com args[1341266186] : ('0.06356172894348543',) no score found for photo 1341266186 We are sending mail with results at report@fotonower.com args[1341265999] : ('0.06356172894348543',) no score found for photo 1341265999 We are sending mail with results at report@fotonower.com args[1341265987] : ('0.06356172894348543',) no score found for photo 1341265987 We are sending mail with results at report@fotonower.com refus_total : 0.06356172894348543 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=21164371 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1341265987,1341321334,1341321328,1341266213,1341266186,1341265999,1341265555,1341265981,1341265971,1341265754,1341265698,1341265629,1341265574,1341265562,1341265975,1341265967,1341266194,1341266207,1341266663,1341321324) Found this number of photos: 20 begin to download photo : 1341265987 begin to download photo : 1341265999 begin to download photo : 1341265698 begin to download photo : 1341265967 download finish for photo 1341265999 begin to download photo : 1341265555 download finish for photo 1341265967 begin to download photo : 1341266194 download finish for photo 1341265987 begin to download photo : 1341321334 download finish for photo 1341265698 begin to download photo : 1341265629 download finish for photo 1341321334 begin to download photo : 1341321328 download finish for photo 1341266194 begin to download photo : 1341266207 download finish for photo 1341265629 begin to download photo : 1341265574 download finish for photo 1341265555 begin to download photo : 1341265981 download finish for photo 1341321328 begin to download photo : 1341266213 download finish for photo 1341266207 begin to download photo : 1341266663 download finish for photo 1341265574 begin to download photo : 1341265562 download finish for photo 1341265981 begin to download photo : 1341265971 download finish for photo 1341266663 begin to download photo : 1341321324 download finish for photo 1341266213 begin to download photo : 1341266186 download finish for photo 1341265562 begin to download photo : 1341265975 download finish for photo 1341321324 download finish for photo 1341266186 download finish for photo 1341265971 begin to download photo : 1341265754 download finish for photo 1341265975 download finish for photo 1341265754 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21164371_08-03-2025_00_24_30.pdf results_Auto_P21164371_08-03-2025_00_24_30.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21164371_08-03-2025_00_24_30.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','21164371','results_Auto_P21164371_08-03-2025_00_24_30.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21164371_08-03-2025_00_24_30.pdf','pdf','','0.82','0.06356172894348543') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1341321334, 1341321328, 1341321324, 1341266663, 1341266213, 1341266207, 1341266194, 1341266186, 1341265999, 1341265987] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341321334', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341321328', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341321324', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266663', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266213', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266207', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266194', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266186', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341265999', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341265987', None, None, None, None, None, '2650228') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.014527082443237305 save_final save missing photos in datou_result : time spend for datou_step_exec : 9.052619218826294 time spend to save output : 0.014836549758911133 total time spend for step 10 : 9.067455768585205 step11:split_time_score Sat Mar 8 00:24:39 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('17', 20),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 06032025 21164371 Nombre de photos uploadées : 20 / 23040 (0%) 06032025 21164371 Nombre de photos taguées (types de déchets): 0 / 20 (0%) 06032025 21164371 Nombre de photos taguées (volume) : 0 / 20 (0%) elapsed_time : load_data_split_time_score 1.6689300537109375e-06 elapsed_time : order_list_meta_photo_and_scores 5.245208740234375e-06 ???????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0011410713195800781 elapsed_time : insert_dashboard_record_day_entry 0.03171277046203613 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164242 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164243 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164244 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164245 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164246 order by id desc limit 1 Qualite : 0.08069052068974839 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21164247_08-03-2025_00_21_28.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164247 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21164247 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164248 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164249 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164250 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164251 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164252 order by id desc limit 1 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21164253_08-03-2025_00_20_54.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164253 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164254 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164255 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164256 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164257 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164258 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164259 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164260 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164261 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164262 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164263 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164264 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164279 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164280 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164281 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164282 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164283 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21132535 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164284 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164302 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164303 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164304 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164305 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131105 order by id desc limit 1 Qualite : 0.1208628538400596 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21131106_07-03-2025_01_05_21.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131106 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21131106 AND mptpi.`type`=3726 To do Qualite : 0.2403201734976397 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21131107_07-03-2025_01_22_22.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131107 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21131107 AND mptpi.`type`=3594 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131108 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164306 order by id desc limit 1 Qualite : 0.23950699235455492 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21131110_07-03-2025_00_45_00.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131110 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21131110 AND mptpi.`type`=3594 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164307 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164315 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164323 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164327 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164328 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164329 order by id desc limit 1 Qualite : 0.11756225525408417 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21164330_08-03-2025_00_23_27.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164330 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21164330 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164337 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164349 order by id desc limit 1 Qualite : 0.24659821273223817 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131864 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21131864 AND mptpi.`type`=3594 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164350 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164351 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164352 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164353 order by id desc limit 1 Qualite : 0.48615912660497385 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21164370_08-03-2025_00_23_27.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164370 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11548 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11556 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11557 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11552 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11552 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11555 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11555 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11580 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11580 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11551 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11550 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11550 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11559 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11551 doesn't seem to be define in the database( WARNING : type of input 3 of step 11550 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11548 doesn't seem to be define in the database( WARNING : type of input 2 of step 11552 doesn't seem to be define in the database( WARNING : output 1 of step 11548 have datatype=2 whereas input 1 of step 11555 have datatype=7 WARNING : type of output 2 of step 11555 doesn't seem to be define in the database( WARNING : type of input 1 of step 11549 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11551 have datatype=10 whereas input 3 of step 11558 have datatype=6 WARNING : type of input 2 of step 11580 doesn't seem to be define in the database( WARNING : output 1 of step 11549 have datatype=7 whereas input 2 of step 11580 have datatype=None WARNING : type of output 3 of step 11580 doesn't seem to be define in the database( WARNING : type of input 1 of step 11551 doesn't seem to be define in the database( WARNING : output 0 of step 11551 have datatype=10 whereas input 0 of step 11586 have datatype=18 WARNING : type of input 5 of step 11558 doesn't seem to be define in the database( WARNING : output 0 of step 11586 have datatype=11 whereas input 5 of step 11558 have datatype=None WARNING : type of output 1 of step 11556 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : type of output 1 of step 11557 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : output 0 of step 11555 have datatype=1 whereas input 0 of step 11549 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21164370 AND mptpi.`type`=4200 To do Qualite : 0.22215740173384166 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21131114_07-03-2025_00_29_29.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21131114 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21131114 AND mptpi.`type`=3594 To do Qualite : 0.06356172894348543 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21164371_08-03-2025_00_24_30.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164371 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21164371 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164378 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164390 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164391 order by id desc limit 1 Qualite : 0.07180205898152511 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21164392_08-03-2025_00_21_52.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164392 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21164392 AND mptpi.`type`=4207 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164393 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164394 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164395 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164396 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164397 order by id desc limit 1 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21164398_08-03-2025_00_24_53.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164398 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164399 order by id desc limit 1 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21164400_08-03-2025_00_24_38.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164400 order by id desc limit 1 Qualite : 0.061715560852036645 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21164401_08-03-2025_00_22_13.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164401 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21164401 AND mptpi.`type`=4207 To do Qualite : 0.09179105009673029 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21164402_08-03-2025_00_25_15.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21164402 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21164402 AND mptpi.`type`=4207 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'06032025': {'nb_upload': 20, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1341321334, 1341321328, 1341321324, 1341266663, 1341266213, 1341266207, 1341266194, 1341266186, 1341265999, 1341265987] Looping around the photos to save general results len do output : 1 /21164371Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341321334', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341321328', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341321324', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266663', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266213', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266207', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266194', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341266186', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341265999', None, None, None, None, None, '2650228') ('3459', None, None, None, None, None, None, None, '2650228') ('3459', '21164371', '1341265987', None, None, None, None, None, '2650228') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.016431331634521484 save_final save missing photos in datou_result : time spend for datou_step_exec : 47.3409686088562 time spend to save output : 0.016684293746948242 total time spend for step 11 : 47.35765290260315 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 138.05user 203.89system 7:03.60elapsed 80%CPU (0avgtext+0avgdata 4072648maxresident)k 1165304inputs+110584outputs (20720major+12486979minor)pagefaults 0swaps