python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3332437 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 12>10 please execute split_portfolio.py -i 21144584 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2648611'] with mtr_portfolio_ids : ['21144584'] and first list_photo_ids : [] new path : /proc/3332437/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 1.6088743209838867 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Fri Mar 7 15:24:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10593 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-03-07 15:24:33.536891: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-07 15:24:33.567197: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-07 15:24:33.569297: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f88a8000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-07 15:24:33.569336: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-07 15:24:33.572536: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-07 15:24:33.805474: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x40ab0b10 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-07 15:24:33.805538: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-07 15:24:33.806826: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-07 15:24:33.807206: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-07 15:24:33.846056: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-07 15:24:33.849334: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-07 15:24:33.850190: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-07 15:24:33.853044: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-07 15:24:33.854180: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-07 15:24:33.859005: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-07 15:24:33.860558: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-07 15:24:33.860639: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-07 15:24:33.861440: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-07 15:24:33.861457: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-07 15:24:33.861467: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-07 15:24:33.862847: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9815 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-03-07 15:24:34.151465: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-07 15:24:34.151595: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-07 15:24:34.151619: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-07 15:24:34.151641: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-07 15:24:34.151661: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-07 15:24:34.151697: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-07 15:24:34.151718: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-07 15:24:34.151738: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-07 15:24:34.153464: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-07 15:24:34.154931: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-07 15:24:34.155020: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-07 15:24:34.155054: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-07 15:24:34.155083: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-07 15:24:34.155111: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-07 15:24:34.155161: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-07 15:24:34.155188: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-07 15:24:34.155212: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-07 15:24:34.157031: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-07 15:24:34.157066: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-07 15:24:34.157077: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-07 15:24:34.157087: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-07 15:24:34.158750: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9815 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-03-07 15:24:44.191764: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-07 15:24:44.389169: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 16 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 18 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 16 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 15 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 14 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 11 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 148.80000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 27 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 25 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 19 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 15 Detection mask done ! Trying to reset tf kernel 3332804 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5304 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 10593 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.006437063217163086 nb_pixel_total : 328481 time to create 1 rle with new method : 0.015043020248413086 length of segment : 553 time for calcul the mask position with numpy : 0.000171661376953125 nb_pixel_total : 4034 time to create 1 rle with old method : 0.004904747009277344 length of segment : 133 time for calcul the mask position with numpy : 0.000362396240234375 nb_pixel_total : 20647 time to create 1 rle with old method : 0.023760318756103516 length of segment : 194 time for calcul the mask position with numpy : 0.0007808208465576172 nb_pixel_total : 53397 time to create 1 rle with old method : 0.0612950325012207 length of segment : 331 time for calcul the mask position with numpy : 0.00025844573974609375 nb_pixel_total : 18395 time to create 1 rle with old method : 0.02025747299194336 length of segment : 141 time for calcul the mask position with numpy : 0.0011572837829589844 nb_pixel_total : 75407 time to create 1 rle with old method : 0.08165836334228516 length of segment : 295 time for calcul the mask position with numpy : 0.001491546630859375 nb_pixel_total : 67617 time to create 1 rle with old method : 0.0723412036895752 length of segment : 326 time for calcul the mask position with numpy : 0.0001404285430908203 nb_pixel_total : 3413 time to create 1 rle with old method : 0.00421595573425293 length of segment : 78 time for calcul the mask position with numpy : 0.0005555152893066406 nb_pixel_total : 23498 time to create 1 rle with old method : 0.026726245880126953 length of segment : 206 time for calcul the mask position with numpy : 7.581710815429688e-05 nb_pixel_total : 2282 time to create 1 rle with old method : 0.002599954605102539 length of segment : 59 time for calcul the mask position with numpy : 8.463859558105469e-05 nb_pixel_total : 3640 time to create 1 rle with old method : 0.004531145095825195 length of segment : 57 time for calcul the mask position with numpy : 0.0004954338073730469 nb_pixel_total : 27912 time to create 1 rle with old method : 0.04546809196472168 length of segment : 279 time for calcul the mask position with numpy : 0.00010824203491210938 nb_pixel_total : 5136 time to create 1 rle with old method : 0.0058367252349853516 length of segment : 106 time for calcul the mask position with numpy : 0.0018916130065917969 nb_pixel_total : 117288 time to create 1 rle with old method : 0.1322317123413086 length of segment : 473 time for calcul the mask position with numpy : 0.0007784366607666016 nb_pixel_total : 56673 time to create 1 rle with old method : 0.06256341934204102 length of segment : 360 time for calcul the mask position with numpy : 0.00014090538024902344 nb_pixel_total : 3754 time to create 1 rle with old method : 0.004656076431274414 length of segment : 99 time for calcul the mask position with numpy : 0.005112886428833008 nb_pixel_total : 302153 time to create 1 rle with new method : 0.016987085342407227 length of segment : 616 time for calcul the mask position with numpy : 8.225440979003906e-05 nb_pixel_total : 1268 time to create 1 rle with old method : 0.0015778541564941406 length of segment : 48 time for calcul the mask position with numpy : 0.0003707408905029297 nb_pixel_total : 30207 time to create 1 rle with old method : 0.03670907020568848 length of segment : 105 time for calcul the mask position with numpy : 7.796287536621094e-05 nb_pixel_total : 2284 time to create 1 rle with old method : 0.002786874771118164 length of segment : 53 time for calcul the mask position with numpy : 0.0010378360748291016 nb_pixel_total : 65428 time to create 1 rle with old method : 0.07228755950927734 length of segment : 505 time for calcul the mask position with numpy : 0.0007801055908203125 nb_pixel_total : 47629 time to create 1 rle with old method : 0.05195879936218262 length of segment : 352 time for calcul the mask position with numpy : 0.00012111663818359375 nb_pixel_total : 5249 time to create 1 rle with old method : 0.005876779556274414 length of segment : 93 time for calcul the mask position with numpy : 0.0002551078796386719 nb_pixel_total : 16780 time to create 1 rle with old method : 0.018007516860961914 length of segment : 122 time for calcul the mask position with numpy : 0.002781391143798828 nb_pixel_total : 167947 time to create 1 rle with new method : 0.011486530303955078 length of segment : 785 time for calcul the mask position with numpy : 0.0007088184356689453 nb_pixel_total : 22926 time to create 1 rle with old method : 0.026023387908935547 length of segment : 311 time for calcul the mask position with numpy : 0.00029349327087402344 nb_pixel_total : 15527 time to create 1 rle with old method : 0.017743349075317383 length of segment : 113 time for calcul the mask position with numpy : 0.00031280517578125 nb_pixel_total : 8172 time to create 1 rle with old method : 0.009513378143310547 length of segment : 254 time for calcul the mask position with numpy : 0.00015854835510253906 nb_pixel_total : 4298 time to create 1 rle with old method : 0.00539398193359375 length of segment : 51 time for calcul the mask position with numpy : 0.00023174285888671875 nb_pixel_total : 8632 time to create 1 rle with old method : 0.010317087173461914 length of segment : 104 time for calcul the mask position with numpy : 9.274482727050781e-05 nb_pixel_total : 2594 time to create 1 rle with old method : 0.0031690597534179688 length of segment : 95 time for calcul the mask position with numpy : 0.0015156269073486328 nb_pixel_total : 122477 time to create 1 rle with old method : 0.13796114921569824 length of segment : 413 time for calcul the mask position with numpy : 9.989738464355469e-05 nb_pixel_total : 3779 time to create 1 rle with old method : 0.004293680191040039 length of segment : 76 time for calcul the mask position with numpy : 0.0005967617034912109 nb_pixel_total : 40432 time to create 1 rle with old method : 0.04415607452392578 length of segment : 289 time for calcul the mask position with numpy : 0.0021495819091796875 nb_pixel_total : 114660 time to create 1 rle with old method : 0.12403106689453125 length of segment : 617 time for calcul the mask position with numpy : 0.0012464523315429688 nb_pixel_total : 56652 time to create 1 rle with old method : 0.061767578125 length of segment : 417 time for calcul the mask position with numpy : 0.00010037422180175781 nb_pixel_total : 3762 time to create 1 rle with old method : 0.00421595573425293 length of segment : 81 time for calcul the mask position with numpy : 0.0002524852752685547 nb_pixel_total : 11799 time to create 1 rle with old method : 0.012855768203735352 length of segment : 174 time for calcul the mask position with numpy : 0.0002732276916503906 nb_pixel_total : 14377 time to create 1 rle with old method : 0.016506195068359375 length of segment : 195 time for calcul the mask position with numpy : 0.0005686283111572266 nb_pixel_total : 34182 time to create 1 rle with old method : 0.03728652000427246 length of segment : 252 time for calcul the mask position with numpy : 0.0028603076934814453 nb_pixel_total : 125758 time to create 1 rle with old method : 0.13646531105041504 length of segment : 665 time for calcul the mask position with numpy : 0.0027914047241210938 nb_pixel_total : 130605 time to create 1 rle with old method : 0.15184259414672852 length of segment : 665 time for calcul the mask position with numpy : 9.107589721679688e-05 nb_pixel_total : 1074 time to create 1 rle with old method : 0.0014789104461669922 length of segment : 62 time for calcul the mask position with numpy : 0.0013005733489990234 nb_pixel_total : 62021 time to create 1 rle with old method : 0.0724947452545166 length of segment : 460 time for calcul the mask position with numpy : 0.0004684925079345703 nb_pixel_total : 9835 time to create 1 rle with old method : 0.011356115341186523 length of segment : 224 time for calcul the mask position with numpy : 0.0017971992492675781 nb_pixel_total : 61030 time to create 1 rle with old method : 0.07294011116027832 length of segment : 297 time for calcul the mask position with numpy : 0.0010075569152832031 nb_pixel_total : 36448 time to create 1 rle with old method : 0.04216909408569336 length of segment : 302 time for calcul the mask position with numpy : 0.004608869552612305 nb_pixel_total : 237576 time to create 1 rle with new method : 0.010869741439819336 length of segment : 474 time for calcul the mask position with numpy : 0.0011982917785644531 nb_pixel_total : 47516 time to create 1 rle with old method : 0.05405783653259277 length of segment : 227 time for calcul the mask position with numpy : 0.0015153884887695312 nb_pixel_total : 28497 time to create 1 rle with old method : 0.03248739242553711 length of segment : 374 time for calcul the mask position with numpy : 0.0004639625549316406 nb_pixel_total : 13536 time to create 1 rle with old method : 0.016469478607177734 length of segment : 157 time for calcul the mask position with numpy : 0.0004413127899169922 nb_pixel_total : 14233 time to create 1 rle with old method : 0.016750574111938477 length of segment : 209 time for calcul the mask position with numpy : 0.0005776882171630859 nb_pixel_total : 14689 time to create 1 rle with old method : 0.016937971115112305 length of segment : 225 time for calcul the mask position with numpy : 0.0023784637451171875 nb_pixel_total : 66796 time to create 1 rle with old method : 0.07702350616455078 length of segment : 363 time for calcul the mask position with numpy : 0.00014972686767578125 nb_pixel_total : 1409 time to create 1 rle with old method : 0.0017278194427490234 length of segment : 60 time for calcul the mask position with numpy : 0.001552581787109375 nb_pixel_total : 45095 time to create 1 rle with old method : 0.05008721351623535 length of segment : 274 time for calcul the mask position with numpy : 0.0011234283447265625 nb_pixel_total : 36546 time to create 1 rle with old method : 0.040729522705078125 length of segment : 327 time for calcul the mask position with numpy : 0.0018723011016845703 nb_pixel_total : 83914 time to create 1 rle with old method : 0.09420228004455566 length of segment : 304 time for calcul the mask position with numpy : 0.0017609596252441406 nb_pixel_total : 87148 time to create 1 rle with old method : 0.0966494083404541 length of segment : 365 time for calcul the mask position with numpy : 0.0012331008911132812 nb_pixel_total : 24856 time to create 1 rle with old method : 0.028472185134887695 length of segment : 239 time for calcul the mask position with numpy : 0.0004582405090332031 nb_pixel_total : 18416 time to create 1 rle with old method : 0.022037029266357422 length of segment : 170 time for calcul the mask position with numpy : 0.0028793811798095703 nb_pixel_total : 133803 time to create 1 rle with old method : 0.14557099342346191 length of segment : 463 time for calcul the mask position with numpy : 0.0030536651611328125 nb_pixel_total : 138372 time to create 1 rle with old method : 0.15323877334594727 length of segment : 453 time for calcul the mask position with numpy : 0.00022220611572265625 nb_pixel_total : 3032 time to create 1 rle with old method : 0.0037021636962890625 length of segment : 77 time for calcul the mask position with numpy : 0.006232500076293945 nb_pixel_total : 255204 time to create 1 rle with new method : 0.017238855361938477 length of segment : 539 time for calcul the mask position with numpy : 0.001890420913696289 nb_pixel_total : 33607 time to create 1 rle with old method : 0.03847336769104004 length of segment : 384 time for calcul the mask position with numpy : 0.001829385757446289 nb_pixel_total : 80485 time to create 1 rle with old method : 0.0925896167755127 length of segment : 419 time for calcul the mask position with numpy : 0.004953622817993164 nb_pixel_total : 171844 time to create 1 rle with new method : 0.013744592666625977 length of segment : 1045 time for calcul the mask position with numpy : 0.0002675056457519531 nb_pixel_total : 2129 time to create 1 rle with old method : 0.0027413368225097656 length of segment : 139 time for calcul the mask position with numpy : 0.0013928413391113281 nb_pixel_total : 42685 time to create 1 rle with old method : 0.04843711853027344 length of segment : 427 time for calcul the mask position with numpy : 0.0019903182983398438 nb_pixel_total : 83239 time to create 1 rle with old method : 0.09456276893615723 length of segment : 449 time for calcul the mask position with numpy : 0.0003094673156738281 nb_pixel_total : 10111 time to create 1 rle with old method : 0.011323928833007812 length of segment : 104 time for calcul the mask position with numpy : 0.0004918575286865234 nb_pixel_total : 15934 time to create 1 rle with old method : 0.01813650131225586 length of segment : 142 time for calcul the mask position with numpy : 0.0009474754333496094 nb_pixel_total : 40766 time to create 1 rle with old method : 0.04655194282531738 length of segment : 195 time for calcul the mask position with numpy : 0.0009284019470214844 nb_pixel_total : 41088 time to create 1 rle with old method : 0.06801676750183105 length of segment : 204 time for calcul the mask position with numpy : 0.0004019737243652344 nb_pixel_total : 6827 time to create 1 rle with old method : 0.011564493179321289 length of segment : 206 time for calcul the mask position with numpy : 0.0018475055694580078 nb_pixel_total : 31281 time to create 1 rle with old method : 0.0507206916809082 length of segment : 352 time for calcul the mask position with numpy : 0.001210927963256836 nb_pixel_total : 23431 time to create 1 rle with old method : 0.026610612869262695 length of segment : 283 time for calcul the mask position with numpy : 0.003849029541015625 nb_pixel_total : 142553 time to create 1 rle with old method : 0.16940879821777344 length of segment : 540 time for calcul the mask position with numpy : 0.003053426742553711 nb_pixel_total : 162521 time to create 1 rle with new method : 0.006429433822631836 length of segment : 480 time for calcul the mask position with numpy : 0.0008409023284912109 nb_pixel_total : 21484 time to create 1 rle with old method : 0.025161027908325195 length of segment : 134 time for calcul the mask position with numpy : 0.0011508464813232422 nb_pixel_total : 34628 time to create 1 rle with old method : 0.04078984260559082 length of segment : 156 time for calcul the mask position with numpy : 0.00047707557678222656 nb_pixel_total : 9878 time to create 1 rle with old method : 0.01520228385925293 length of segment : 137 time for calcul the mask position with numpy : 0.002674102783203125 nb_pixel_total : 57258 time to create 1 rle with old method : 0.07049036026000977 length of segment : 326 time for calcul the mask position with numpy : 0.0002753734588623047 nb_pixel_total : 2379 time to create 1 rle with old method : 0.004106760025024414 length of segment : 86 time for calcul the mask position with numpy : 0.00036978721618652344 nb_pixel_total : 6457 time to create 1 rle with old method : 0.008966684341430664 length of segment : 87 time for calcul the mask position with numpy : 0.00020265579223632812 nb_pixel_total : 3245 time to create 1 rle with old method : 0.004076480865478516 length of segment : 56 time for calcul the mask position with numpy : 0.0015447139739990234 nb_pixel_total : 47097 time to create 1 rle with old method : 0.053887128829956055 length of segment : 348 time for calcul the mask position with numpy : 0.0011134147644042969 nb_pixel_total : 34216 time to create 1 rle with old method : 0.040422916412353516 length of segment : 267 time for calcul the mask position with numpy : 0.0015006065368652344 nb_pixel_total : 29910 time to create 1 rle with old method : 0.034609079360961914 length of segment : 305 time for calcul the mask position with numpy : 0.008666753768920898 nb_pixel_total : 166429 time to create 1 rle with new method : 0.019016265869140625 length of segment : 699 time for calcul the mask position with numpy : 0.0012087821960449219 nb_pixel_total : 40153 time to create 1 rle with old method : 0.04780864715576172 length of segment : 168 time for calcul the mask position with numpy : 0.00019311904907226562 nb_pixel_total : 3366 time to create 1 rle with old method : 0.004154205322265625 length of segment : 69 time for calcul the mask position with numpy : 0.0008492469787597656 nb_pixel_total : 36947 time to create 1 rle with old method : 0.04347801208496094 length of segment : 263 time for calcul the mask position with numpy : 0.00030517578125 nb_pixel_total : 6359 time to create 1 rle with old method : 0.009290218353271484 length of segment : 109 time for calcul the mask position with numpy : 0.0018334388732910156 nb_pixel_total : 50543 time to create 1 rle with old method : 0.07023382186889648 length of segment : 240 time for calcul the mask position with numpy : 0.003141641616821289 nb_pixel_total : 55913 time to create 1 rle with old method : 0.06695127487182617 length of segment : 438 time for calcul the mask position with numpy : 0.0021898746490478516 nb_pixel_total : 78126 time to create 1 rle with old method : 0.08780932426452637 length of segment : 301 time for calcul the mask position with numpy : 0.002810239791870117 nb_pixel_total : 109586 time to create 1 rle with old method : 0.1266183853149414 length of segment : 387 time for calcul the mask position with numpy : 0.0012030601501464844 nb_pixel_total : 40468 time to create 1 rle with old method : 0.04732942581176758 length of segment : 195 time for calcul the mask position with numpy : 0.001049041748046875 nb_pixel_total : 23589 time to create 1 rle with old method : 0.07557439804077148 length of segment : 330 time for calcul the mask position with numpy : 0.0006339550018310547 nb_pixel_total : 26262 time to create 1 rle with old method : 0.033849477767944336 length of segment : 146 time for calcul the mask position with numpy : 0.004122257232666016 nb_pixel_total : 144514 time to create 1 rle with old method : 0.2585105895996094 length of segment : 605 time for calcul the mask position with numpy : 0.004153013229370117 nb_pixel_total : 158875 time to create 1 rle with new method : 0.012719154357910156 length of segment : 552 time for calcul the mask position with numpy : 0.000759124755859375 nb_pixel_total : 25945 time to create 1 rle with old method : 0.04006028175354004 length of segment : 147 time for calcul the mask position with numpy : 0.0020270347595214844 nb_pixel_total : 59538 time to create 1 rle with old method : 0.07779932022094727 length of segment : 323 time for calcul the mask position with numpy : 0.0012345314025878906 nb_pixel_total : 33247 time to create 1 rle with old method : 0.05970001220703125 length of segment : 254 time for calcul the mask position with numpy : 0.001032114028930664 nb_pixel_total : 25143 time to create 1 rle with old method : 0.03716850280761719 length of segment : 163 time for calcul the mask position with numpy : 0.0712440013885498 nb_pixel_total : 1463491 time to create 1 rle with new method : 0.57305908203125 length of segment : 3426 time for calcul the mask position with numpy : 0.0007407665252685547 nb_pixel_total : 19368 time to create 1 rle with old method : 0.030389785766601562 length of segment : 179 time for calcul the mask position with numpy : 0.0021741390228271484 nb_pixel_total : 84236 time to create 1 rle with old method : 0.09494137763977051 length of segment : 379 time for calcul the mask position with numpy : 0.00025963783264160156 nb_pixel_total : 7627 time to create 1 rle with old method : 0.008800506591796875 length of segment : 106 time for calcul the mask position with numpy : 0.00302886962890625 nb_pixel_total : 187025 time to create 1 rle with new method : 0.007575035095214844 length of segment : 523 time for calcul the mask position with numpy : 0.0016775131225585938 nb_pixel_total : 96120 time to create 1 rle with old method : 0.10842156410217285 length of segment : 472 time for calcul the mask position with numpy : 0.0066509246826171875 nb_pixel_total : 252786 time to create 1 rle with new method : 0.013835430145263672 length of segment : 1451 time for calcul the mask position with numpy : 0.010221004486083984 nb_pixel_total : 450765 time to create 1 rle with new method : 0.022060871124267578 length of segment : 936 time for calcul the mask position with numpy : 0.0026078224182128906 nb_pixel_total : 152238 time to create 1 rle with new method : 0.006875753402709961 length of segment : 573 time for calcul the mask position with numpy : 0.004284858703613281 nb_pixel_total : 115451 time to create 1 rle with old method : 0.1350860595703125 length of segment : 846 time for calcul the mask position with numpy : 0.00017833709716796875 nb_pixel_total : 6216 time to create 1 rle with old method : 0.007565736770629883 length of segment : 74 time for calcul the mask position with numpy : 0.0014040470123291016 nb_pixel_total : 54134 time to create 1 rle with old method : 0.06394100189208984 length of segment : 180 time for calcul the mask position with numpy : 0.0009839534759521484 nb_pixel_total : 55277 time to create 1 rle with old method : 0.06590032577514648 length of segment : 494 time for calcul the mask position with numpy : 0.0003879070281982422 nb_pixel_total : 22420 time to create 1 rle with old method : 0.028548240661621094 length of segment : 184 time for calcul the mask position with numpy : 0.002416372299194336 nb_pixel_total : 134976 time to create 1 rle with old method : 0.16564559936523438 length of segment : 536 time for calcul the mask position with numpy : 0.00019931793212890625 nb_pixel_total : 5715 time to create 1 rle with old method : 0.006729841232299805 length of segment : 95 time for calcul the mask position with numpy : 0.0016880035400390625 nb_pixel_total : 119435 time to create 1 rle with old method : 0.13303303718566895 length of segment : 484 time for calcul the mask position with numpy : 0.002002716064453125 nb_pixel_total : 122763 time to create 1 rle with old method : 0.1579570770263672 length of segment : 330 time for calcul the mask position with numpy : 0.0003447532653808594 nb_pixel_total : 19481 time to create 1 rle with old method : 0.0227048397064209 length of segment : 151 time for calcul the mask position with numpy : 0.0001392364501953125 nb_pixel_total : 5090 time to create 1 rle with old method : 0.0061304569244384766 length of segment : 68 time for calcul the mask position with numpy : 0.002947092056274414 nb_pixel_total : 144391 time to create 1 rle with old method : 0.16018915176391602 length of segment : 519 time for calcul the mask position with numpy : 0.01398015022277832 nb_pixel_total : 672466 time to create 1 rle with new method : 0.033757925033569336 length of segment : 965 time for calcul the mask position with numpy : 0.00043082237243652344 nb_pixel_total : 14083 time to create 1 rle with old method : 0.01657414436340332 length of segment : 101 time for calcul the mask position with numpy : 0.0024967193603515625 nb_pixel_total : 109117 time to create 1 rle with old method : 0.12169599533081055 length of segment : 606 time for calcul the mask position with numpy : 0.0007984638214111328 nb_pixel_total : 26881 time to create 1 rle with old method : 0.032019853591918945 length of segment : 262 time for calcul the mask position with numpy : 0.0008041858673095703 nb_pixel_total : 43918 time to create 1 rle with old method : 0.05043983459472656 length of segment : 161 time for calcul the mask position with numpy : 0.0004942417144775391 nb_pixel_total : 12774 time to create 1 rle with old method : 0.014928340911865234 length of segment : 138 time for calcul the mask position with numpy : 0.005040884017944336 nb_pixel_total : 159233 time to create 1 rle with new method : 0.010033607482910156 length of segment : 873 time for calcul the mask position with numpy : 0.001634836196899414 nb_pixel_total : 93892 time to create 1 rle with old method : 0.130845308303833 length of segment : 374 time for calcul the mask position with numpy : 0.0010347366333007812 nb_pixel_total : 42597 time to create 1 rle with old method : 0.04904985427856445 length of segment : 322 time for calcul the mask position with numpy : 0.0021703243255615234 nb_pixel_total : 107063 time to create 1 rle with old method : 0.11888504028320312 length of segment : 424 time for calcul the mask position with numpy : 0.00016951560974121094 nb_pixel_total : 4128 time to create 1 rle with old method : 0.00528264045715332 length of segment : 49 time for calcul the mask position with numpy : 0.0008435249328613281 nb_pixel_total : 60240 time to create 1 rle with old method : 0.06800270080566406 length of segment : 205 time for calcul the mask position with numpy : 0.024483442306518555 nb_pixel_total : 1014899 time to create 1 rle with new method : 0.32538652420043945 length of segment : 2476 time for calcul the mask position with numpy : 0.0002834796905517578 nb_pixel_total : 4314 time to create 1 rle with old method : 0.012551546096801758 length of segment : 72 time for calcul the mask position with numpy : 0.0026612281799316406 nb_pixel_total : 116302 time to create 1 rle with old method : 0.1388247013092041 length of segment : 439 time for calcul the mask position with numpy : 0.0001628398895263672 nb_pixel_total : 1844 time to create 1 rle with old method : 0.004328489303588867 length of segment : 41 time for calcul the mask position with numpy : 0.02628612518310547 nb_pixel_total : 881665 time to create 1 rle with new method : 0.2606637477874756 length of segment : 1902 time for calcul the mask position with numpy : 0.0005364418029785156 nb_pixel_total : 26355 time to create 1 rle with old method : 0.030419349670410156 length of segment : 186 time for calcul the mask position with numpy : 0.0005116462707519531 nb_pixel_total : 15471 time to create 1 rle with old method : 0.018101215362548828 length of segment : 176 time for calcul the mask position with numpy : 0.0004820823669433594 nb_pixel_total : 22516 time to create 1 rle with old method : 0.026539325714111328 length of segment : 234 time for calcul the mask position with numpy : 0.0015811920166015625 nb_pixel_total : 51849 time to create 1 rle with old method : 0.060007572174072266 length of segment : 321 time for calcul the mask position with numpy : 0.00032830238342285156 nb_pixel_total : 8782 time to create 1 rle with old method : 0.010564565658569336 length of segment : 123 time for calcul the mask position with numpy : 0.028450727462768555 nb_pixel_total : 839864 time to create 1 rle with new method : 0.0904548168182373 length of segment : 1583 time for calcul the mask position with numpy : 0.0019059181213378906 nb_pixel_total : 100152 time to create 1 rle with old method : 0.11251068115234375 length of segment : 393 time for calcul the mask position with numpy : 0.10905313491821289 nb_pixel_total : 1964516 time to create 1 rle with new method : 0.7661778926849365 length of segment : 4011 time for calcul the mask position with numpy : 0.0021898746490478516 nb_pixel_total : 143397 time to create 1 rle with old method : 0.15980172157287598 length of segment : 382 time for calcul the mask position with numpy : 0.010254383087158203 nb_pixel_total : 534689 time to create 1 rle with new method : 0.047396183013916016 length of segment : 1203 time for calcul the mask position with numpy : 0.0030972957611083984 nb_pixel_total : 217399 time to create 1 rle with new method : 0.0075223445892333984 length of segment : 764 time for calcul the mask position with numpy : 0.0001494884490966797 nb_pixel_total : 3337 time to create 1 rle with old method : 0.004645586013793945 length of segment : 84 time for calcul the mask position with numpy : 0.00020170211791992188 nb_pixel_total : 2958 time to create 1 rle with old method : 0.003478527069091797 length of segment : 179 time spent for convertir_results : 21.41246771812439 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 159 chid ids of type : 3663 Number RLEs to save : 61501 save missing photos in datou_result : time spend for datou_step_exec : 67.79521918296814 time spend to save output : 3.5685839653015137 total time spend for step 1 : 71.36380314826965 step2:crop_condition Fri Mar 7 15:25:42 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 159 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 19 About to insert : list_path_to_insert length 19 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 13 About to insert : list_path_to_insert length 13 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 56 About to insert : list_path_to_insert length 56 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1341451594, 1341451526, 1341451521, 1341451517, 1341451513, 1341451510, 1341451505, 1341451455, 1341451450, 1341451445] Looping around the photos to save general results len do output : 105 /-3697387604Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387611Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387608Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387606Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387628Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387627Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387646Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387641Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387638Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387652Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387650Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387651Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387668Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387665Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387664Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387666Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387673Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387687Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387703Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387595Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387633Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387654Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387662Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387697Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387681Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387733Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387598Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387599Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387634Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387695Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387686Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387691Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387696Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387698Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387715Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387721Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387726Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387745Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387747Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387601Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387596Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387593Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387609Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387614Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387618Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387620Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387607Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387616Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387636Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387632Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387624Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387626Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387630Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387649Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387642Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387640Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387648Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387660Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387655Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387657Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387653Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387659Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387658Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387670Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387669Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387672Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387667Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387663Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387683Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387694Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387689Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387684Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387680Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387682Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387690Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387702Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387706Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387707Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387700Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387718Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387709Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387705Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387713Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387717Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387714Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387719Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387728Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387723Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387729Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387724Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387731Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387742Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387739Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387743Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387741Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387631Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387676Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387704Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387710Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387699Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387671Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387711Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387740Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387613Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3697387716Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451594', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451526', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451521', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451517', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451513', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451510', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451505', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451455', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451450', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451445', None, None, None, None, None, '2648611') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 325 time used for this insertion : 0.029810428619384766 save_final save missing photos in datou_result : time spend for datou_step_exec : 46.14202094078064 time spend to save output : 0.03231048583984375 total time spend for step 2 : 46.17433142662048 step3:thcl Fri Mar 7 15:26:28 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.013438224792480469 time to convert the images to numpy array : 0.08790159225463867 time to import caffe and check if the image exist : 0.011159181594848633 time to convert the images to numpy array : 0.13414597511291504 time to import caffe and check if the image exist : 0.013715505599975586 time to convert the images to numpy array : 0.14359378814697266 time to import caffe and check if the image exist : 0.01788783073425293 time to convert the images to numpy array : 0.31734251976013184 time to import caffe and check if the image exist : 0.012657403945922852 time to convert the images to numpy array : 0.3265268802642822 time to import caffe and check if the image exist : 0.014713048934936523 time to convert the images to numpy array : 0.35166168212890625 time to import caffe and check if the image exist : 0.016036033630371094 time to convert the images to numpy array : 0.36023497581481934 time to import caffe and check if the image exist : 0.015323877334594727 time to convert the images to numpy array : 0.47723388671875 time to import caffe and check if the image exist : 0.013122797012329102 time to convert the images to numpy array : 0.4855663776397705 time to import caffe and check if the image exist : 0.010170459747314453 time to convert the images to numpy array : 0.49705004692077637 total time to convert the images to numpy array : 0.8076176643371582 list photo_ids error: [] list photo_ids correct : [-3697387699, -3697387671, -3697387711, -3697387740, -3697387613, -3697387716, -3697387604, -3697387611, -3697387608, -3697387606, -3697387628, -3697387627, -3697387646, -3697387641, -3697387638, -3697387652, -3697387650, -3697387618, -3697387620, -3697387607, -3697387616, -3697387636, -3697387632, -3697387624, -3697387626, -3697387630, -3697387649, -3697387642, -3697387651, -3697387668, -3697387665, -3697387664, -3697387666, -3697387673, -3697387687, -3697387703, -3697387595, -3697387633, -3697387654, -3697387667, -3697387663, -3697387683, -3697387694, -3697387689, -3697387684, -3697387680, -3697387682, -3697387690, -3697387702, -3697387706, -3697387729, -3697387724, -3697387731, -3697387742, -3697387739, -3697387743, -3697387741, -3697387631, -3697387676, -3697387704, -3697387710, -3697387662, -3697387697, -3697387681, -3697387733, -3697387598, -3697387599, -3697387634, -3697387695, -3697387686, -3697387691, -3697387696, -3697387640, -3697387648, -3697387660, -3697387655, -3697387657, -3697387653, -3697387659, -3697387658, -3697387670, -3697387669, -3697387672, -3697387707, -3697387700, -3697387718, -3697387709, -3697387705, -3697387713, -3697387717, -3697387714, -3697387719, -3697387728, -3697387723, -3697387698, -3697387715, -3697387721, -3697387726, -3697387745, -3697387747, -3697387601, -3697387596, -3697387593, -3697387609, -3697387614] number of photos to traite : 105 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10372 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 9802 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.3311948776245117 time used to do the prediction : 0.22231411933898926 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 9352 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 9352 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.4655625820159912 time used to do the prediction : 0.21111202239990234 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 1.9073486328125e-05 save missing photos in datou_result : time spend for datou_step_exec : 10.932007074356079 time spend to save output : 0.0006854534149169922 total time spend for step 3 : 10.932692527770996 step4:merge_mask_thcl_custom Fri Mar 7 15:26:39 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 159 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1341451594, 1341451526, 1341451521, 1341451517, 1341451513, 1341451510, 1341451505, 1341451455, 1341451450, 1341451445] Looping around the photos to save general results len do output : 10 /1341451594Didn't retrieve data .Didn't retrieve data . /1341451526Didn't retrieve data .Didn't retrieve data . /1341451521Didn't retrieve data .Didn't retrieve data . /1341451517Didn't retrieve data .Didn't retrieve data . /1341451513Didn't retrieve data .Didn't retrieve data . /1341451510Didn't retrieve data .Didn't retrieve data . /1341451505Didn't retrieve data .Didn't retrieve data . /1341451455Didn't retrieve data .Didn't retrieve data . /1341451450Didn't retrieve data .Didn't retrieve data . /1341451445Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451594', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451526', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451521', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451517', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451513', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451510', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451505', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451455', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451450', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451445', None, None, None, None, None, '2648611') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.01373744010925293 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.06501984596252441 time spend to save output : 0.014333486557006836 total time spend for step 4 : 0.07935333251953125 step5:rle_unique_nms_with_priority Fri Mar 7 15:26:39 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 11 nb_hashtags : 5 time to prepare the origin masks : 8.043087720870972 time for calcul the mask position with numpy : 0.5949654579162598 nb_pixel_total : 6451246 time to create 1 rle with new method : 0.3297743797302246 time for calcul the mask position with numpy : 0.025497913360595703 nb_pixel_total : 3628 time to create 1 rle with old method : 0.004225254058837891 time for calcul the mask position with numpy : 0.02844977378845215 nb_pixel_total : 67337 time to create 1 rle with old method : 0.08199930191040039 time for calcul the mask position with numpy : 0.03370547294616699 nb_pixel_total : 2281 time to create 1 rle with old method : 0.0025510787963867188 time for calcul the mask position with numpy : 0.0348505973815918 nb_pixel_total : 53223 time to create 1 rle with old method : 0.058165550231933594 time for calcul the mask position with numpy : 0.03794431686401367 nb_pixel_total : 327698 time to create 1 rle with new method : 1.174328327178955 time for calcul the mask position with numpy : 0.03144693374633789 nb_pixel_total : 75035 time to create 1 rle with old method : 0.08333277702331543 time for calcul the mask position with numpy : 0.03429365158081055 nb_pixel_total : 3411 time to create 1 rle with old method : 0.003824949264526367 time for calcul the mask position with numpy : 0.03567361831665039 nb_pixel_total : 18352 time to create 1 rle with old method : 0.022994518280029297 time for calcul the mask position with numpy : 0.026950836181640625 nb_pixel_total : 23400 time to create 1 rle with old method : 0.02612757682800293 time for calcul the mask position with numpy : 0.027720928192138672 nb_pixel_total : 20592 time to create 1 rle with old method : 0.026003122329711914 time for calcul the mask position with numpy : 0.02419281005859375 nb_pixel_total : 4037 time to create 1 rle with old method : 0.004517078399658203 create new chi : 2.820344924926758 time to delete rle : 0.013880491256713867 batch 1 Loaded 12 chid ids of type : 3726 Number RLEs to save : 6870 TO DO : save crop sub photo not yet done ! save time : 0.43627023696899414 nb_obj : 16 nb_hashtags : 6 time to prepare the origin masks : 8.640697956085205 time for calcul the mask position with numpy : 0.7279767990112305 nb_pixel_total : 6333546 time to create 1 rle with new method : 0.443026065826416 time for calcul the mask position with numpy : 0.03694772720336914 nb_pixel_total : 65202 time to create 1 rle with old method : 0.06952834129333496 time for calcul the mask position with numpy : 0.03495526313781738 nb_pixel_total : 56597 time to create 1 rle with old method : 0.06340384483337402 time for calcul the mask position with numpy : 0.03906130790710449 nb_pixel_total : 4227 time to create 1 rle with old method : 0.006989717483520508 time for calcul the mask position with numpy : 0.03946518898010254 nb_pixel_total : 16689 time to create 1 rle with old method : 0.01896357536315918 time for calcul the mask position with numpy : 0.03385663032531738 nb_pixel_total : 8127 time to create 1 rle with old method : 0.008927345275878906 time for calcul the mask position with numpy : 0.03444790840148926 nb_pixel_total : 22888 time to create 1 rle with old method : 0.025289297103881836 time for calcul the mask position with numpy : 0.03542327880859375 nb_pixel_total : 5231 time to create 1 rle with old method : 0.005945444107055664 time for calcul the mask position with numpy : 0.03380846977233887 nb_pixel_total : 2265 time to create 1 rle with old method : 0.0024657249450683594 time for calcul the mask position with numpy : 0.03335094451904297 nb_pixel_total : 3739 time to create 1 rle with old method : 0.004263401031494141 time for calcul the mask position with numpy : 0.03327345848083496 nb_pixel_total : 1261 time to create 1 rle with old method : 0.0014667510986328125 time for calcul the mask position with numpy : 0.032592058181762695 nb_pixel_total : 27820 time to create 1 rle with old method : 0.03321123123168945 time for calcul the mask position with numpy : 0.03268599510192871 nb_pixel_total : 118260 time to create 1 rle with old method : 0.12838411331176758 time for calcul the mask position with numpy : 0.0338442325592041 nb_pixel_total : 47419 time to create 1 rle with old method : 0.05433940887451172 time for calcul the mask position with numpy : 0.034537315368652344 nb_pixel_total : 30155 time to create 1 rle with old method : 0.035703182220458984 time for calcul the mask position with numpy : 0.03386878967285156 nb_pixel_total : 5114 time to create 1 rle with old method : 0.0058078765869140625 time for calcul the mask position with numpy : 0.03675246238708496 nb_pixel_total : 301700 time to create 1 rle with new method : 0.5382835865020752 create new chi : 2.803645610809326 time to delete rle : 0.0018329620361328125 batch 1 Loaded 17 chid ids of type : 3726 Number RLEs to save : 9770 TO DO : save crop sub photo not yet done ! save time : 0.5678243637084961 nb_obj : 15 nb_hashtags : 6 time to prepare the origin masks : 9.26450490951538 time for calcul the mask position with numpy : 0.5117127895355225 nb_pixel_total : 6612967 time to create 1 rle with new method : 0.44177746772766113 time for calcul the mask position with numpy : 0.03409600257873535 nb_pixel_total : 5742 time to create 1 rle with old method : 0.006784200668334961 time for calcul the mask position with numpy : 0.033574819564819336 nb_pixel_total : 11761 time to create 1 rle with old method : 0.012880086898803711 time for calcul the mask position with numpy : 0.033550262451171875 nb_pixel_total : 11821 time to create 1 rle with old method : 0.013089418411254883 time for calcul the mask position with numpy : 0.03216838836669922 nb_pixel_total : 34042 time to create 1 rle with old method : 0.03733944892883301 time for calcul the mask position with numpy : 0.03226613998413086 nb_pixel_total : 1063 time to create 1 rle with old method : 0.001191854476928711 time for calcul the mask position with numpy : 0.02161097526550293 nb_pixel_total : 40227 time to create 1 rle with old method : 0.04370403289794922 time for calcul the mask position with numpy : 0.0248260498046875 nb_pixel_total : 3746 time to create 1 rle with old method : 0.005911827087402344 time for calcul the mask position with numpy : 0.025098562240600586 nb_pixel_total : 122420 time to create 1 rle with old method : 0.14638614654541016 time for calcul the mask position with numpy : 0.021837949752807617 nb_pixel_total : 114653 time to create 1 rle with old method : 0.12547993659973145 time for calcul the mask position with numpy : 0.021201372146606445 nb_pixel_total : 14265 time to create 1 rle with old method : 0.015715837478637695 time for calcul the mask position with numpy : 0.021959543228149414 nb_pixel_total : 3775 time to create 1 rle with old method : 0.004132747650146484 time for calcul the mask position with numpy : 0.03339505195617676 nb_pixel_total : 6015 time to create 1 rle with old method : 0.0070912837982177734 time for calcul the mask position with numpy : 0.03365492820739746 nb_pixel_total : 56601 time to create 1 rle with old method : 0.06264734268188477 time for calcul the mask position with numpy : 0.033071041107177734 nb_pixel_total : 2585 time to create 1 rle with old method : 0.0029528141021728516 time for calcul the mask position with numpy : 0.031141281127929688 nb_pixel_total : 8557 time to create 1 rle with old method : 0.009226799011230469 create new chi : 1.9218904972076416 time to delete rle : 0.0010068416595458984 batch 1 Loaded 16 chid ids of type : 3726 Number RLEs to save : 10317 TO DO : save crop sub photo not yet done ! save time : 0.6078755855560303 nb_obj : 13 nb_hashtags : 4 time to prepare the origin masks : 7.795903921127319 time for calcul the mask position with numpy : 0.5744900703430176 nb_pixel_total : 6435888 time to create 1 rle with new method : 0.5236082077026367 time for calcul the mask position with numpy : 0.0208132266998291 nb_pixel_total : 66377 time to create 1 rle with old method : 0.06933212280273438 time for calcul the mask position with numpy : 0.020078659057617188 nb_pixel_total : 47445 time to create 1 rle with old method : 0.04991793632507324 time for calcul the mask position with numpy : 0.02101922035217285 nb_pixel_total : 44965 time to create 1 rle with old method : 0.04864501953125 time for calcul the mask position with numpy : 0.022522687911987305 nb_pixel_total : 240719 time to create 1 rle with new method : 0.45540642738342285 time for calcul the mask position with numpy : 0.02872014045715332 nb_pixel_total : 1401 time to create 1 rle with old method : 0.0026750564575195312 time for calcul the mask position with numpy : 0.02937150001525879 nb_pixel_total : 35925 time to create 1 rle with old method : 0.04675722122192383 time for calcul the mask position with numpy : 0.024196386337280273 nb_pixel_total : 60787 time to create 1 rle with old method : 0.06894183158874512 time for calcul the mask position with numpy : 0.02395033836364746 nb_pixel_total : 14657 time to create 1 rle with old method : 0.01633453369140625 time for calcul the mask position with numpy : 0.02416253089904785 nb_pixel_total : 14131 time to create 1 rle with old method : 0.02179551124572754 time for calcul the mask position with numpy : 0.02477550506591797 nb_pixel_total : 13445 time to create 1 rle with old method : 0.02118372917175293 time for calcul the mask position with numpy : 0.029769182205200195 nb_pixel_total : 28556 time to create 1 rle with old method : 0.04531669616699219 time for calcul the mask position with numpy : 0.02968454360961914 nb_pixel_total : 36125 time to create 1 rle with old method : 0.052208900451660156 time for calcul the mask position with numpy : 0.024166584014892578 nb_pixel_total : 9819 time to create 1 rle with old method : 0.011040449142456055 create new chi : 2.404498815536499 time to delete rle : 0.0009298324584960938 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 9008 TO DO : save crop sub photo not yet done ! save time : 0.5337474346160889 nb_obj : 12 nb_hashtags : 6 time to prepare the origin masks : 8.615267992019653 time for calcul the mask position with numpy : 0.5615768432617188 nb_pixel_total : 6010812 time to create 1 rle with new method : 0.7585203647613525 time for calcul the mask position with numpy : 0.04864382743835449 nb_pixel_total : 18737 time to create 1 rle with old method : 0.030295133590698242 time for calcul the mask position with numpy : 0.03287506103515625 nb_pixel_total : 138209 time to create 1 rle with old method : 0.1518409252166748 time for calcul the mask position with numpy : 0.022971153259277344 nb_pixel_total : 24748 time to create 1 rle with old method : 0.027856826782226562 time for calcul the mask position with numpy : 0.025323867797851562 nb_pixel_total : 9765 time to create 1 rle with old method : 0.012322664260864258 time for calcul the mask position with numpy : 0.024245738983154297 nb_pixel_total : 3019 time to create 1 rle with old method : 0.0032968521118164062 time for calcul the mask position with numpy : 0.023114919662475586 nb_pixel_total : 87314 time to create 1 rle with old method : 0.09523534774780273 time for calcul the mask position with numpy : 0.02236175537109375 nb_pixel_total : 133718 time to create 1 rle with old method : 0.1472029685974121 time for calcul the mask position with numpy : 0.021802425384521484 nb_pixel_total : 171556 time to create 1 rle with new method : 1.1557719707489014 time for calcul the mask position with numpy : 0.021317481994628906 nb_pixel_total : 80366 time to create 1 rle with old method : 0.08826470375061035 time for calcul the mask position with numpy : 0.021325111389160156 nb_pixel_total : 33493 time to create 1 rle with old method : 0.036734819412231445 time for calcul the mask position with numpy : 0.023011207580566406 nb_pixel_total : 83756 time to create 1 rle with old method : 0.09339570999145508 time for calcul the mask position with numpy : 0.024589061737060547 nb_pixel_total : 254747 time to create 1 rle with new method : 0.7659652233123779 create new chi : 4.3326475620269775 time to delete rle : 0.0015532970428466797 batch 1 Loaded 13 chid ids of type : 3726 Number RLEs to save : 11716 TO DO : save crop sub photo not yet done ! save time : 1.319495677947998 nb_obj : 10 nb_hashtags : 4 time to prepare the origin masks : 8.010955095291138 time for calcul the mask position with numpy : 0.5683157444000244 nb_pixel_total : 6532693 time to create 1 rle with new method : 0.78891921043396 time for calcul the mask position with numpy : 0.023996829986572266 nb_pixel_total : 2017 time to create 1 rle with old method : 0.0038275718688964844 time for calcul the mask position with numpy : 0.028404951095581055 nb_pixel_total : 162287 time to create 1 rle with new method : 2.2437188625335693 time for calcul the mask position with numpy : 0.0210111141204834 nb_pixel_total : 31132 time to create 1 rle with old method : 0.03483223915100098 time for calcul the mask position with numpy : 0.02408599853515625 nb_pixel_total : 142271 time to create 1 rle with old method : 0.15501642227172852 time for calcul the mask position with numpy : 0.02123260498046875 nb_pixel_total : 15849 time to create 1 rle with old method : 0.01693892478942871 time for calcul the mask position with numpy : 0.019417524337768555 nb_pixel_total : 23345 time to create 1 rle with old method : 0.02461981773376465 time for calcul the mask position with numpy : 0.02054286003112793 nb_pixel_total : 40607 time to create 1 rle with old method : 0.04310727119445801 time for calcul the mask position with numpy : 0.020256996154785156 nb_pixel_total : 83171 time to create 1 rle with old method : 0.08719968795776367 time for calcul the mask position with numpy : 0.020317792892456055 nb_pixel_total : 6783 time to create 1 rle with old method : 0.0077702999114990234 time for calcul the mask position with numpy : 0.02116870880126953 nb_pixel_total : 10085 time to create 1 rle with old method : 0.011932134628295898 create new chi : 4.272841930389404 time to delete rle : 0.0012905597686767578 batch 1 Loaded 11 chid ids of type : 3726 Number RLEs to save : 7966 TO DO : save crop sub photo not yet done ! save time : 0.6279706954956055 nb_obj : 25 nb_hashtags : 7 time to prepare the origin masks : 13.85826063156128 time for calcul the mask position with numpy : 0.48685145378112793 nb_pixel_total : 6004989 time to create 1 rle with new method : 1.0198004245758057 time for calcul the mask position with numpy : 0.03788566589355469 nb_pixel_total : 47034 time to create 1 rle with old method : 0.05134010314941406 time for calcul the mask position with numpy : 0.034697771072387695 nb_pixel_total : 29844 time to create 1 rle with old method : 0.03315401077270508 time for calcul the mask position with numpy : 0.03568267822265625 nb_pixel_total : 120875 time to create 1 rle with old method : 0.13558650016784668 time for calcul the mask position with numpy : 0.03388643264770508 nb_pixel_total : 29824 time to create 1 rle with old method : 0.03223443031311035 time for calcul the mask position with numpy : 0.03380846977233887 nb_pixel_total : 77883 time to create 1 rle with old method : 0.16893219947814941 time for calcul the mask position with numpy : 0.03481650352478027 nb_pixel_total : 3454 time to create 1 rle with old method : 0.006193399429321289 time for calcul the mask position with numpy : 0.0390315055847168 nb_pixel_total : 4214 time to create 1 rle with old method : 0.006983757019042969 time for calcul the mask position with numpy : 0.03849363327026367 nb_pixel_total : 3829 time to create 1 rle with old method : 0.004771232604980469 time for calcul the mask position with numpy : 0.03991270065307617 nb_pixel_total : 34077 time to create 1 rle with old method : 0.05107831954956055 time for calcul the mask position with numpy : 0.034473419189453125 nb_pixel_total : 270 time to create 1 rle with old method : 0.0005381107330322266 time for calcul the mask position with numpy : 0.034291744232177734 nb_pixel_total : 50279 time to create 1 rle with old method : 0.05473780632019043 time for calcul the mask position with numpy : 0.03235793113708496 nb_pixel_total : 26135 time to create 1 rle with old method : 0.028682470321655273 time for calcul the mask position with numpy : 0.033914804458618164 nb_pixel_total : 3343 time to create 1 rle with old method : 0.003633260726928711 time for calcul the mask position with numpy : 0.03352713584899902 nb_pixel_total : 6345 time to create 1 rle with old method : 0.006848812103271484 time for calcul the mask position with numpy : 0.033864498138427734 nb_pixel_total : 56963 time to create 1 rle with old method : 0.0665597915649414 time for calcul the mask position with numpy : 0.03992486000061035 nb_pixel_total : 39931 time to create 1 rle with old method : 0.04348254203796387 time for calcul the mask position with numpy : 0.033994197845458984 nb_pixel_total : 3223 time to create 1 rle with old method : 0.003670930862426758 time for calcul the mask position with numpy : 0.037203311920166016 nb_pixel_total : 165170 time to create 1 rle with new method : 0.43731141090393066 time for calcul the mask position with numpy : 0.03351783752441406 nb_pixel_total : 6435 time to create 1 rle with old method : 0.006708621978759766 time for calcul the mask position with numpy : 0.0327153205871582 nb_pixel_total : 2371 time to create 1 rle with old method : 0.0026798248291015625 time for calcul the mask position with numpy : 0.035416364669799805 nb_pixel_total : 9839 time to create 1 rle with old method : 0.010714530944824219 time for calcul the mask position with numpy : 0.03490257263183594 nb_pixel_total : 21425 time to create 1 rle with old method : 0.023414134979248047 time for calcul the mask position with numpy : 0.03434133529663086 nb_pixel_total : 34489 time to create 1 rle with old method : 0.03721117973327637 time for calcul the mask position with numpy : 0.033388376235961914 nb_pixel_total : 109322 time to create 1 rle with old method : 0.11714434623718262 time for calcul the mask position with numpy : 0.03525257110595703 nb_pixel_total : 158677 time to create 1 rle with new method : 0.470001220703125 create new chi : 4.273271322250366 time to delete rle : 0.0013539791107177734 batch 1 Loaded 26 chid ids of type : 3726 Number RLEs to save : 13789 TO DO : save crop sub photo not yet done ! save time : 0.8041892051696777 nb_obj : 21 nb_hashtags : 6 time to prepare the origin masks : 12.98094654083252 time for calcul the mask position with numpy : 0.40073418617248535 nb_pixel_total : 4099980 time to create 1 rle with new method : 0.37562060356140137 time for calcul the mask position with numpy : 0.03731203079223633 nb_pixel_total : 1472 time to create 1 rle with old method : 0.0018277168273925781 time for calcul the mask position with numpy : 0.03513765335083008 nb_pixel_total : 416 time to create 1 rle with old method : 0.0007340908050537109 time for calcul the mask position with numpy : 0.0342860221862793 nb_pixel_total : 55425 time to create 1 rle with old method : 0.09092283248901367 time for calcul the mask position with numpy : 0.04953336715698242 nb_pixel_total : 116920 time to create 1 rle with old method : 0.13413786888122559 time for calcul the mask position with numpy : 0.03469681739807129 nb_pixel_total : 4974 time to create 1 rle with old method : 0.0058176517486572266 time for calcul the mask position with numpy : 0.03512454032897949 nb_pixel_total : 53849 time to create 1 rle with old method : 0.060320138931274414 time for calcul the mask position with numpy : 0.03639960289001465 nb_pixel_total : 19248 time to create 1 rle with old method : 0.021561384201049805 time for calcul the mask position with numpy : 0.030178070068359375 nb_pixel_total : 5696 time to create 1 rle with old method : 0.006436586380004883 time for calcul the mask position with numpy : 0.021842241287231445 nb_pixel_total : 13877 time to create 1 rle with old method : 0.01595616340637207 time for calcul the mask position with numpy : 0.02450108528137207 nb_pixel_total : 78984 time to create 1 rle with old method : 0.0876777172088623 time for calcul the mask position with numpy : 0.023056745529174805 nb_pixel_total : 25040 time to create 1 rle with old method : 0.027926206588745117 time for calcul the mask position with numpy : 0.022628068923950195 nb_pixel_total : 122451 time to create 1 rle with old method : 0.13396573066711426 time for calcul the mask position with numpy : 0.021166086196899414 nb_pixel_total : 5763 time to create 1 rle with old method : 0.006482839584350586 time for calcul the mask position with numpy : 0.025193214416503906 nb_pixel_total : 450574 time to create 1 rle with new method : 1.076711654663086 time for calcul the mask position with numpy : 0.02787327766418457 nb_pixel_total : 151591 time to create 1 rle with new method : 0.7788870334625244 time for calcul the mask position with numpy : 0.026114940643310547 nb_pixel_total : 115672 time to create 1 rle with old method : 0.1276073455810547 time for calcul the mask position with numpy : 0.02907848358154297 nb_pixel_total : 7593 time to create 1 rle with old method : 0.008732080459594727 time for calcul the mask position with numpy : 0.05556225776672363 nb_pixel_total : 1448225 time to create 1 rle with new method : 0.7796924114227295 time for calcul the mask position with numpy : 0.03630661964416504 nb_pixel_total : 826 time to create 1 rle with old method : 0.0015058517456054688 time for calcul the mask position with numpy : 0.0388941764831543 nb_pixel_total : 252214 time to create 1 rle with new method : 0.44985389709472656 time for calcul the mask position with numpy : 0.03592491149902344 nb_pixel_total : 19450 time to create 1 rle with old method : 0.02176189422607422 create new chi : 5.439798593521118 time to delete rle : 0.002932310104370117 batch 1 Loaded 22 chid ids of type : 3726 Number RLEs to save : 22248 TO DO : save crop sub photo not yet done ! save time : 1.3931257724761963 nb_obj : 16 nb_hashtags : 6 time to prepare the origin masks : 7.813261985778809 time for calcul the mask position with numpy : 0.39250707626342773 nb_pixel_total : 4573807 time to create 1 rle with new method : 0.7880351543426514 time for calcul the mask position with numpy : 0.03472590446472168 nb_pixel_total : 158505 time to create 1 rle with new method : 1.1835110187530518 time for calcul the mask position with numpy : 0.03488588333129883 nb_pixel_total : 17722 time to create 1 rle with old method : 0.01951313018798828 time for calcul the mask position with numpy : 0.033316850662231445 nb_pixel_total : 13994 time to create 1 rle with old method : 0.015041828155517578 time for calcul the mask position with numpy : 0.033002614974975586 nb_pixel_total : 9644 time to create 1 rle with old method : 0.010465860366821289 time for calcul the mask position with numpy : 0.03074049949645996 nb_pixel_total : 4276 time to create 1 rle with old method : 0.004390716552734375 time for calcul the mask position with numpy : 0.03689146041870117 nb_pixel_total : 1015567 time to create 1 rle with new method : 0.5349762439727783 time for calcul the mask position with numpy : 0.021144628524780273 nb_pixel_total : 108965 time to create 1 rle with old method : 0.11402511596679688 time for calcul the mask position with numpy : 0.03286409378051758 nb_pixel_total : 107047 time to create 1 rle with old method : 0.11348342895507812 time for calcul the mask position with numpy : 0.03269338607788086 nb_pixel_total : 4081 time to create 1 rle with old method : 0.0045659542083740234 time for calcul the mask position with numpy : 0.0336613655090332 nb_pixel_total : 145395 time to create 1 rle with old method : 0.15287160873413086 time for calcul the mask position with numpy : 0.036261558532714844 nb_pixel_total : 42263 time to create 1 rle with old method : 0.04508543014526367 time for calcul the mask position with numpy : 0.031163930892944336 nb_pixel_total : 12710 time to create 1 rle with old method : 0.01314997673034668 time for calcul the mask position with numpy : 0.026924848556518555 nb_pixel_total : 43762 time to create 1 rle with old method : 0.04603934288024902 time for calcul the mask position with numpy : 0.03346538543701172 nb_pixel_total : 26549 time to create 1 rle with old method : 0.028631210327148438 time for calcul the mask position with numpy : 0.03706645965576172 nb_pixel_total : 672192 time to create 1 rle with new method : 0.37673139572143555 time for calcul the mask position with numpy : 0.030974388122558594 nb_pixel_total : 93761 time to create 1 rle with old method : 0.09876561164855957 create new chi : 4.569046974182129 time to delete rle : 0.0014712810516357422 batch 1 Loaded 17 chid ids of type : 3726 Number RLEs to save : 17771 TO DO : save crop sub photo not yet done ! save time : 8.206096410751343 nb_obj : 13 nb_hashtags : 5 time to prepare the origin masks : 8.24095106124878 time for calcul the mask position with numpy : 0.4790036678314209 nb_pixel_total : 3990132 time to create 1 rle with new method : 1.0371699333190918 time for calcul the mask position with numpy : 0.034207820892333984 nb_pixel_total : 8759 time to create 1 rle with old method : 0.00955963134765625 time for calcul the mask position with numpy : 0.03445744514465332 nb_pixel_total : 13887 time to create 1 rle with old method : 0.015149354934692383 time for calcul the mask position with numpy : 0.03410816192626953 nb_pixel_total : 2433 time to create 1 rle with old method : 0.0027544498443603516 time for calcul the mask position with numpy : 0.0368039608001709 nb_pixel_total : 210072 time to create 1 rle with new method : 0.9953269958496094 time for calcul the mask position with numpy : 0.03720664978027344 nb_pixel_total : 50068 time to create 1 rle with old method : 0.07372474670410156 time for calcul the mask position with numpy : 0.03679537773132324 nb_pixel_total : 167728 time to create 1 rle with new method : 0.3075125217437744 time for calcul the mask position with numpy : 0.04006600379943848 nb_pixel_total : 598985 time to create 1 rle with new method : 0.6308560371398926 time for calcul the mask position with numpy : 0.035106658935546875 nb_pixel_total : 99856 time to create 1 rle with old method : 0.11956906318664551 time for calcul the mask position with numpy : 0.03159737586975098 nb_pixel_total : 838745 time to create 1 rle with new method : 1.6060657501220703 time for calcul the mask position with numpy : 0.024195194244384766 nb_pixel_total : 26288 time to create 1 rle with old method : 0.029168128967285156 time for calcul the mask position with numpy : 0.02695751190185547 nb_pixel_total : 143150 time to create 1 rle with old method : 0.18893885612487793 time for calcul the mask position with numpy : 0.04415607452392578 nb_pixel_total : 877713 time to create 1 rle with new method : 0.623140811920166 time for calcul the mask position with numpy : 0.03713655471801758 nb_pixel_total : 22424 time to create 1 rle with old method : 0.0249330997467041 create new chi : 6.751873016357422 time to delete rle : 0.0023212432861328125 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 19349 TO DO : save crop sub photo not yet done ! save time : 1.2337172031402588 map_output_result : {1341451594: (0.0, 'Should be the crop_list due to order', 0), 1341451526: (0.0, 'Should be the crop_list due to order', 0), 1341451521: (0.0, 'Should be the crop_list due to order', 0), 1341451517: (0.0, 'Should be the crop_list due to order', 0), 1341451513: (0.0, 'Should be the crop_list due to order', 0), 1341451510: (0.0, 'Should be the crop_list due to order', 0), 1341451505: (0.0, 'Should be the crop_list due to order', 0), 1341451455: (0.0, 'Should be the crop_list due to order', 0), 1341451450: (0.0, 'Should be the crop_list due to order', 0), 1341451445: (0.0, 'Should be the crop_list due to order', 0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1341451594, 1341451526, 1341451521, 1341451517, 1341451513, 1341451510, 1341451505, 1341451455, 1341451450, 1341451445] Looping around the photos to save general results len do output : 10 /1341451594.Didn't retrieve data . /1341451526.Didn't retrieve data . /1341451521.Didn't retrieve data . /1341451517.Didn't retrieve data . /1341451513.Didn't retrieve data . /1341451510.Didn't retrieve data . /1341451505.Didn't retrieve data . /1341451455.Didn't retrieve data . /1341451450.Didn't retrieve data . /1341451445.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451594', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451526', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451521', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451517', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451513', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451510', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451505', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451455', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451450', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451445', None, None, None, None, None, '2648611') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013110876083374023 save_final save missing photos in datou_result : time spend for datou_step_exec : 149.74303030967712 time spend to save output : 0.013778924942016602 total time spend for step 5 : 149.75680923461914 step6:crop_condition Fri Mar 7 15:29:09 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 162 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! About to upload 8 photos upload in portfolio : 4869462 init cache_photo without model_param we have 8 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741357753_3332437 we have uploaded 8 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.2518491744995117 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! About to upload 8 photos upload in portfolio : 4869462 init cache_photo without model_param we have 8 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741357759_3332437 we have uploaded 8 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.483203649520874 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! About to upload 9 photos upload in portfolio : 4869462 init cache_photo without model_param we have 9 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741357765_3332437 we have uploaded 9 photos in the portfolio 4869462 time of upload the photos Elapsed time : 5.433160781860352 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741357772_3332437 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.092790126800537 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! About to upload 8 photos upload in portfolio : 4869462 init cache_photo without model_param we have 8 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741357779_3332437 we have uploaded 8 photos in the portfolio 4869462 time of upload the photos Elapsed time : 4.649488925933838 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741357786_3332437 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.288249969482422 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741357790_3332437 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.6475188732147217 we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1341451594, 1341451526, 1341451521, 1341451517, 1341451513, 1341451510, 1341451505, 1341451455, 1341451450, 1341451445] Looping around the photos to save general results len do output : 41 /1341477862Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341477864Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341477865Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341477866Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341477868Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341477870Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341477871Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341477872Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341477891Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341477892Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341477893Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341477894Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341477895Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341477896Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341477897Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341477898Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341477933Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341477934Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341477935Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341477937Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341477938Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341477939Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341477941Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341477942Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341477945Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341477966Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341477967Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341477968Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341478029Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341478030Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341478032Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341478033Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341478034Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341478037Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341478038Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341478039Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341478070Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341478071Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341478073Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341478074Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1341478091Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451594', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451526', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451521', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451517', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451513', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451510', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451505', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451455', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451450', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451445', None, None, None, None, None, '2648611') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 133 time used for this insertion : 0.4366333484649658 save_final save missing photos in datou_result : time spend for datou_step_exec : 42.262239933013916 time spend to save output : 0.4378674030303955 total time spend for step 6 : 42.70010733604431 step7:ventilate_hashtags_in_portfolio Fri Mar 7 15:29:51 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 21144584 get user id for portfolio 21144584 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21144584 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('kraft','Carton_gris','metal','plastique','flou','autre_refus','mal_croppe','environnement','papier','cartonnette','Carton_brun','Teint_Dans_La_Masse')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21144584 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('kraft','Carton_gris','metal','plastique','flou','autre_refus','mal_croppe','environnement','papier','cartonnette','Carton_brun','Teint_Dans_La_Masse')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21144584 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('kraft','Carton_gris','metal','plastique','flou','autre_refus','mal_croppe','environnement','papier','cartonnette','Carton_brun','Teint_Dans_La_Masse')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/21144701,21144702,21144703,21144704,21144705,21144706,21144707,21144708,21144709,21144710,21144711,21144712?tags=kraft,Carton_gris,metal,plastique,flou,autre_refus,mal_croppe,environnement,papier,cartonnette,Carton_brun,Teint_Dans_La_Masse Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1341451594, 1341451526, 1341451521, 1341451517, 1341451513, 1341451510, 1341451505, 1341451455, 1341451450, 1341451445] Looping around the photos to save general results len do output : 1 /21144584. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451594', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451526', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451521', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451517', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451513', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451510', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451505', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451455', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451450', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451445', None, None, None, None, None, '2648611') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.013867616653442383 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.7166240215301514 time spend to save output : 0.014156579971313477 total time spend for step 7 : 1.7307806015014648 step8:final Fri Mar 7 15:29:53 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1341451594: ('0.08495526296092404',), 1341451526: ('0.08495526296092404',), 1341451521: ('0.08495526296092404',), 1341451517: ('0.08495526296092404',), 1341451513: ('0.08495526296092404',), 1341451510: ('0.08495526296092404',), 1341451505: ('0.08495526296092404',), 1341451455: ('0.08495526296092404',), 1341451450: ('0.08495526296092404',), 1341451445: ('0.08495526296092404',)} new output for save of step final : {1341451594: ('0.08495526296092404',), 1341451526: ('0.08495526296092404',), 1341451521: ('0.08495526296092404',), 1341451517: ('0.08495526296092404',), 1341451513: ('0.08495526296092404',), 1341451510: ('0.08495526296092404',), 1341451505: ('0.08495526296092404',), 1341451455: ('0.08495526296092404',), 1341451450: ('0.08495526296092404',), 1341451445: ('0.08495526296092404',)} [1341451594, 1341451526, 1341451521, 1341451517, 1341451513, 1341451510, 1341451505, 1341451455, 1341451450, 1341451445] Looping around the photos to save general results len do output : 10 /1341451594.Didn't retrieve data . /1341451526.Didn't retrieve data . /1341451521.Didn't retrieve data . /1341451517.Didn't retrieve data . /1341451513.Didn't retrieve data . /1341451510.Didn't retrieve data . /1341451505.Didn't retrieve data . /1341451455.Didn't retrieve data . /1341451450.Didn't retrieve data . /1341451445.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451594', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451526', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451521', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451517', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451513', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451510', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451505', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451455', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451450', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451445', None, None, None, None, None, '2648611') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.014038324356079102 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.10157966613769531 time spend to save output : 0.01453709602355957 total time spend for step 8 : 0.11611676216125488 step9:velours_tree Fri Mar 7 15:29:53 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.059003591537475586 time spend to save output : 4.0531158447265625e-05 total time spend for step 9 : 0.05904412269592285 step10:send_mail_cod Fri Mar 7 15:29:53 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P21144584_07-03-2025_15_29_53.pdf 21144701 change filename to text .imagette211447011741357793 21144703 imagette211447031741357793 21144704 change filename to text .change filename to text .change filename to text .change filename to text .imagette211447041741357793 21144705 imagette211447051741357794 21144706 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette211447061741357794 21144707 imagette211447071741357794 21144710 change filename to text .change filename to text .change filename to text .imagette211447101741357794 21144702 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette211447021741357794 21144711 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette211447111741357795 21144712 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette211447121741357796 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=21144584 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/21144701,21144702,21144703,21144704,21144705,21144706,21144707,21144708,21144709,21144710,21144711,21144712?tags=kraft,Carton_gris,metal,plastique,flou,autre_refus,mal_croppe,environnement,papier,cartonnette,Carton_brun,Teint_Dans_La_Masse your option no_mail is active, we will not send the real mail to your client args[1341451594] : ('0.08495526296092404',) no score found for photo 1341451594 We are sending mail with results at report@fotonower.com args[1341451526] : ('0.08495526296092404',) no score found for photo 1341451526 We are sending mail with results at report@fotonower.com args[1341451521] : ('0.08495526296092404',) no score found for photo 1341451521 We are sending mail with results at report@fotonower.com args[1341451517] : ('0.08495526296092404',) no score found for photo 1341451517 We are sending mail with results at report@fotonower.com args[1341451513] : ('0.08495526296092404',) no score found for photo 1341451513 We are sending mail with results at report@fotonower.com args[1341451510] : ('0.08495526296092404',) no score found for photo 1341451510 We are sending mail with results at report@fotonower.com args[1341451505] : ('0.08495526296092404',) no score found for photo 1341451505 We are sending mail with results at report@fotonower.com args[1341451455] : ('0.08495526296092404',) no score found for photo 1341451455 We are sending mail with results at report@fotonower.com args[1341451450] : ('0.08495526296092404',) no score found for photo 1341451450 We are sending mail with results at report@fotonower.com args[1341451445] : ('0.08495526296092404',) no score found for photo 1341451445 We are sending mail with results at report@fotonower.com refus_total : 0.08495526296092404 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=21144584 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1341451505,1341451510,1341451513,1341451517,1341451526,1341451594,1341451441,1341451421,1341451445,1341451450,1341451455,1341451521) Found this number of photos: 12 begin to download photo : 1341451505 begin to download photo : 1341451517 begin to download photo : 1341451441 begin to download photo : 1341451450 download finish for photo 1341451505 begin to download photo : 1341451510 download finish for photo 1341451450 begin to download photo : 1341451455 download finish for photo 1341451441 begin to download photo : 1341451421 download finish for photo 1341451517 begin to download photo : 1341451526 download finish for photo 1341451510 begin to download photo : 1341451513 download finish for photo 1341451526 begin to download photo : 1341451594 download finish for photo 1341451421 begin to download photo : 1341451445 download finish for photo 1341451455 begin to download photo : 1341451521 download finish for photo 1341451594 download finish for photo 1341451445 download finish for photo 1341451521 download finish for photo 1341451513 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21144584_07-03-2025_15_29_53.pdf results_Auto_P21144584_07-03-2025_15_29_53.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21144584_07-03-2025_15_29_53.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','21144584','results_Auto_P21144584_07-03-2025_15_29_53.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21144584_07-03-2025_15_29_53.pdf','pdf','','0.71','0.08495526296092404') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1341451594, 1341451526, 1341451521, 1341451517, 1341451513, 1341451510, 1341451505, 1341451455, 1341451450, 1341451445] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451594', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451526', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451521', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451517', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451513', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451510', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451505', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451455', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451450', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451445', None, None, None, None, None, '2648611') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.01172947883605957 save_final save missing photos in datou_result : time spend for datou_step_exec : 5.688031911849976 time spend to save output : 0.011979103088378906 total time spend for step 10 : 5.7000110149383545 step11:split_time_score Fri Mar 7 15:29:59 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('12', 12),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 07032025 21144584 Nombre de photos uploadées : 12 / 23040 (0%) 07032025 21144584 Nombre de photos taguées (types de déchets): 0 / 12 (0%) 07032025 21144584 Nombre de photos taguées (volume) : 0 / 12 (0%) elapsed_time : load_data_split_time_score 6.198883056640625e-06 elapsed_time : order_list_meta_photo_and_scores 6.4373016357421875e-06 ???????????? elapsed_time : fill_and_build_computed_from_old_data 0.0039865970611572266 elapsed_time : insert_dashboard_record_day_entry 0.02302837371826172 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21137165 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21137166 order by id desc limit 1 Qualite : 0.06901780564613035 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21137167_07-03-2025_09_30_13.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21137167 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21137167 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21137168 order by id desc limit 1 Qualite : 0.31035864228901067 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21137169_07-03-2025_09_45_24.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21137169 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11548 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11556 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11557 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11552 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11552 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11555 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11555 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11580 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11580 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11551 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11550 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11550 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11559 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11551 doesn't seem to be define in the database( WARNING : type of input 3 of step 11550 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11548 doesn't seem to be define in the database( WARNING : type of input 2 of step 11552 doesn't seem to be define in the database( WARNING : output 1 of step 11548 have datatype=2 whereas input 1 of step 11555 have datatype=7 WARNING : type of output 2 of step 11555 doesn't seem to be define in the database( WARNING : type of input 1 of step 11549 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11551 have datatype=10 whereas input 3 of step 11558 have datatype=6 WARNING : type of input 2 of step 11580 doesn't seem to be define in the database( WARNING : output 1 of step 11549 have datatype=7 whereas input 2 of step 11580 have datatype=None WARNING : type of output 3 of step 11580 doesn't seem to be define in the database( WARNING : type of input 1 of step 11551 doesn't seem to be define in the database( WARNING : output 0 of step 11551 have datatype=10 whereas input 0 of step 11586 have datatype=18 WARNING : type of input 5 of step 11558 doesn't seem to be define in the database( WARNING : output 0 of step 11586 have datatype=11 whereas input 5 of step 11558 have datatype=None WARNING : type of output 1 of step 11556 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : type of output 1 of step 11557 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : output 0 of step 11555 have datatype=1 whereas input 0 of step 11549 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21137169 AND mptpi.`type`=4200 To do Qualite : 0.12029958453427352 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21137170_07-03-2025_10_12_00.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21137170 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11560 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11567 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11567 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11563 crop_condition is not consistent : 4 used against 2 in the step definition ! Step 11563 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11564 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11564 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11565 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11573 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11573 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11568 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11566 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11566 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 1 of step 11560 have datatype=2 whereas input 1 of step 11564 have datatype=7 WARNING : type of output 2 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11565 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11567 doesn't seem to be define in the database( WARNING : type of input 3 of step 11563 doesn't seem to be define in the database( WARNING : type of output 3 of step 11564 doesn't seem to be define in the database( WARNING : type of input 1 of step 11568 doesn't seem to be define in the database( WARNING : type of output 1 of step 11568 doesn't seem to be define in the database( WARNING : type of input 3 of step 11566 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11570 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of input 2 of step 11563 doesn't seem to be define in the database( WARNING : output 0 of step 11569 have datatype=6 whereas input 2 of step 11563 have datatype=None WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11570 doesn't seem to be define in the database( WARNING : type of output 2 of step 11560 doesn't seem to be define in the database( WARNING : type of input 1 of step 11569 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11568 have datatype=10 whereas input 3 of step 11571 have datatype=6 WARNING : type of input 2 of step 11573 doesn't seem to be define in the database( WARNING : output 1 of step 11565 have datatype=7 whereas input 2 of step 11573 have datatype=None WARNING : type of output 3 of step 11573 doesn't seem to be define in the database( WARNING : type of input 3 of step 11568 doesn't seem to be define in the database( WARNING : output 0 of step 11568 have datatype=10 whereas input 0 of step 11587 have datatype=18 WARNING : type of input 5 of step 11571 doesn't seem to be define in the database( WARNING : output 0 of step 11587 have datatype=11 whereas input 5 of step 11571 have datatype=None WARNING : output 0 of step 11564 have datatype=1 whereas input 0 of step 11565 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21137170 AND mptpi.`type`=3327 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21144562 order by id desc limit 1 Qualite : 0.07653079042292898 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21139330_07-03-2025_15_04_36.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21139330 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21139330 AND mptpi.`type`=4207 To do Qualite : 0.20875828340595504 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21139331_07-03-2025_11_39_16.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21139331 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21139331 AND mptpi.`type`=3594 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21139332 order by id desc limit 1 Qualite : 0.26713232457334796 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21139333_07-03-2025_11_10_50.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21139333 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21139333 AND mptpi.`type`=3594 To do Qualite : 0.10974511151393432 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21139334_07-03-2025_11_15_54.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21139334 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21139334 AND mptpi.`type`=3594 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21144563 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21144564 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21144566 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21144567 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21144568 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21144569 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21144582 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21144583 order by id desc limit 1 Qualite : 0.08495526296092404 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21144584_07-03-2025_15_29_53.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21144584 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21144584 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21144585 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21144586 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21144587 order by id desc limit 1 Qualite : 0.10470139090045394 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21144588_07-03-2025_15_28_32.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21144588 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21144588 AND mptpi.`type`=4207 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21144589 order by id desc limit 1 NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'07032025': {'nb_upload': 12, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1341451594, 1341451526, 1341451521, 1341451517, 1341451513, 1341451510, 1341451505, 1341451455, 1341451450, 1341451445] Looping around the photos to save general results len do output : 1 /21144584Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451594', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451526', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451521', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451517', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451513', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451510', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451505', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451455', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451450', None, None, None, None, None, '2648611') ('3459', None, None, None, None, None, None, None, '2648611') ('3459', '21144584', '1341451445', None, None, None, None, None, '2648611') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.012928962707519531 save_final save missing photos in datou_result : time spend for datou_step_exec : 13.764450550079346 time spend to save output : 0.013148069381713867 total time spend for step 11 : 13.77759861946106 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 163.19user 137.60system 5:47.64elapsed 86%CPU (0avgtext+0avgdata 4422336maxresident)k 36560inputs+123288outputs (510major+14473461minor)pagefaults 0swaps