python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3350784 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 20>10 please execute split_portfolio.py -i 21421446 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2668799'] with mtr_portfolio_ids : ['21421446'] and first list_photo_ids : [] new path : /proc/3350784/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 1.8868536949157715 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Fri Mar 14 11:22:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10814 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-03-14 11:22:32.522354: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-14 11:22:32.547151: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-14 11:22:32.549574: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7faa68000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-14 11:22:32.549605: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-14 11:22:32.553794: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-14 11:22:32.793696: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3d03a6f0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-14 11:22:32.793742: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-14 11:22:32.794649: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-14 11:22:32.795055: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-14 11:22:32.797748: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-14 11:22:32.800302: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-14 11:22:32.800710: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-14 11:22:32.804143: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-14 11:22:32.805408: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-14 11:22:32.811077: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-14 11:22:32.812794: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-14 11:22:32.812915: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-14 11:22:32.813757: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-14 11:22:32.813775: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-14 11:22:32.813785: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-14 11:22:32.815207: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10023 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-03-14 11:22:33.182860: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-14 11:22:33.183062: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-14 11:22:33.183088: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-14 11:22:33.183106: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-14 11:22:33.183124: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-14 11:22:33.183142: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-14 11:22:33.183160: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-14 11:22:33.183179: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-14 11:22:33.185274: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-14 11:22:33.189297: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-14 11:22:33.189504: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-14 11:22:33.189534: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-14 11:22:33.189558: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-14 11:22:33.189580: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-14 11:22:33.189606: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-14 11:22:33.189628: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-14 11:22:33.189651: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-14 11:22:33.191554: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-14 11:22:33.191620: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-14 11:22:33.191636: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-14 11:22:33.191647: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-14 11:22:33.193805: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10023 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-03-14 11:22:47.053716: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-14 11:22:47.371410: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 6 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 9 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 30 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.02358 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 20 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 14 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 11 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 13 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 17 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 19 Detection mask done ! Trying to reset tf kernel 3351320 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 1748 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 7037 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.3348548412322998 nb_pixel_total : 4552759 time to create 1 rle with new method : 0.46410369873046875 length of segment : 4148 time for calcul the mask position with numpy : 0.0007102489471435547 nb_pixel_total : 44382 time to create 1 rle with old method : 0.0496220588684082 length of segment : 493 time for calcul the mask position with numpy : 0.001300811767578125 nb_pixel_total : 92524 time to create 1 rle with old method : 0.10672831535339355 length of segment : 550 time for calcul the mask position with numpy : 0.0020182132720947266 nb_pixel_total : 108189 time to create 1 rle with old method : 0.117218017578125 length of segment : 576 time for calcul the mask position with numpy : 0.0019817352294921875 nb_pixel_total : 110145 time to create 1 rle with old method : 0.12830328941345215 length of segment : 530 time for calcul the mask position with numpy : 0.00034618377685546875 nb_pixel_total : 19605 time to create 1 rle with old method : 0.021814346313476562 length of segment : 240 time for calcul the mask position with numpy : 0.26389598846435547 nb_pixel_total : 2828769 time to create 1 rle with new method : 0.5383753776550293 length of segment : 3349 time for calcul the mask position with numpy : 0.001645803451538086 nb_pixel_total : 78666 time to create 1 rle with old method : 0.0885918140411377 length of segment : 478 time for calcul the mask position with numpy : 0.004655122756958008 nb_pixel_total : 313319 time to create 1 rle with new method : 0.01032710075378418 length of segment : 549 time for calcul the mask position with numpy : 0.005202054977416992 nb_pixel_total : 333193 time to create 1 rle with new method : 0.1390209197998047 length of segment : 722 time for calcul the mask position with numpy : 0.020789623260498047 nb_pixel_total : 594395 time to create 1 rle with new method : 0.26015615463256836 length of segment : 969 time for calcul the mask position with numpy : 0.015624761581420898 nb_pixel_total : 769517 time to create 1 rle with new method : 0.08556008338928223 length of segment : 1165 time for calcul the mask position with numpy : 0.014992475509643555 nb_pixel_total : 633703 time to create 1 rle with new method : 0.25274109840393066 length of segment : 961 time for calcul the mask position with numpy : 0.01606130599975586 nb_pixel_total : 757136 time to create 1 rle with new method : 0.05109119415283203 length of segment : 1415 time for calcul the mask position with numpy : 0.02490091323852539 nb_pixel_total : 1312250 time to create 1 rle with new method : 0.16925644874572754 length of segment : 1928 time for calcul the mask position with numpy : 0.1469261646270752 nb_pixel_total : 3046724 time to create 1 rle with new method : 0.5584547519683838 length of segment : 3745 time for calcul the mask position with numpy : 0.006658792495727539 nb_pixel_total : 318308 time to create 1 rle with new method : 0.012416839599609375 length of segment : 593 time for calcul the mask position with numpy : 0.0017910003662109375 nb_pixel_total : 64765 time to create 1 rle with old method : 0.07550215721130371 length of segment : 372 time for calcul the mask position with numpy : 0.012856006622314453 nb_pixel_total : 686431 time to create 1 rle with new method : 0.04856085777282715 length of segment : 905 time for calcul the mask position with numpy : 0.017454147338867188 nb_pixel_total : 795195 time to create 1 rle with new method : 0.18783259391784668 length of segment : 1438 time for calcul the mask position with numpy : 0.009870529174804688 nb_pixel_total : 322081 time to create 1 rle with new method : 0.022859573364257812 length of segment : 729 time for calcul the mask position with numpy : 0.005071163177490234 nb_pixel_total : 114644 time to create 1 rle with old method : 0.1383800506591797 length of segment : 328 time for calcul the mask position with numpy : 0.002691984176635742 nb_pixel_total : 74633 time to create 1 rle with old method : 0.08530235290527344 length of segment : 362 time for calcul the mask position with numpy : 0.0032100677490234375 nb_pixel_total : 75922 time to create 1 rle with old method : 0.08732032775878906 length of segment : 376 time for calcul the mask position with numpy : 0.0006513595581054688 nb_pixel_total : 9017 time to create 1 rle with old method : 0.010830163955688477 length of segment : 138 time for calcul the mask position with numpy : 0.000431060791015625 nb_pixel_total : 7539 time to create 1 rle with old method : 0.00905156135559082 length of segment : 149 time for calcul the mask position with numpy : 0.006535053253173828 nb_pixel_total : 162611 time to create 1 rle with new method : 0.010653972625732422 length of segment : 572 time for calcul the mask position with numpy : 0.0011298656463623047 nb_pixel_total : 16877 time to create 1 rle with old method : 0.020284414291381836 length of segment : 144 time for calcul the mask position with numpy : 0.0015778541564941406 nb_pixel_total : 38027 time to create 1 rle with old method : 0.043511390686035156 length of segment : 243 time for calcul the mask position with numpy : 0.00017023086547851562 nb_pixel_total : 1544 time to create 1 rle with old method : 0.0021109580993652344 length of segment : 50 time for calcul the mask position with numpy : 0.00027632713317871094 nb_pixel_total : 8156 time to create 1 rle with old method : 0.009799718856811523 length of segment : 126 time for calcul the mask position with numpy : 0.004906415939331055 nb_pixel_total : 110739 time to create 1 rle with old method : 0.1320786476135254 length of segment : 542 time for calcul the mask position with numpy : 0.0004699230194091797 nb_pixel_total : 3268 time to create 1 rle with old method : 0.004323005676269531 length of segment : 89 time for calcul the mask position with numpy : 0.0036880970001220703 nb_pixel_total : 82686 time to create 1 rle with old method : 0.09311938285827637 length of segment : 512 time for calcul the mask position with numpy : 0.0033416748046875 nb_pixel_total : 105980 time to create 1 rle with old method : 0.11904788017272949 length of segment : 639 time for calcul the mask position with numpy : 0.0020601749420166016 nb_pixel_total : 77343 time to create 1 rle with old method : 0.08750343322753906 length of segment : 381 time for calcul the mask position with numpy : 0.0009098052978515625 nb_pixel_total : 18979 time to create 1 rle with old method : 0.022162199020385742 length of segment : 238 time for calcul the mask position with numpy : 0.0014805793762207031 nb_pixel_total : 33336 time to create 1 rle with old method : 0.03920388221740723 length of segment : 262 time for calcul the mask position with numpy : 0.0036237239837646484 nb_pixel_total : 54360 time to create 1 rle with old method : 0.06458711624145508 length of segment : 361 time for calcul the mask position with numpy : 0.002093791961669922 nb_pixel_total : 40734 time to create 1 rle with old method : 0.04825735092163086 length of segment : 559 time for calcul the mask position with numpy : 0.0011944770812988281 nb_pixel_total : 32933 time to create 1 rle with old method : 0.03933882713317871 length of segment : 234 time for calcul the mask position with numpy : 0.0015587806701660156 nb_pixel_total : 44373 time to create 1 rle with old method : 0.05315732955932617 length of segment : 194 time for calcul the mask position with numpy : 0.003885984420776367 nb_pixel_total : 116522 time to create 1 rle with old method : 0.1367485523223877 length of segment : 372 time for calcul the mask position with numpy : 0.0010104179382324219 nb_pixel_total : 35866 time to create 1 rle with old method : 0.05291247367858887 length of segment : 256 time for calcul the mask position with numpy : 0.0003097057342529297 nb_pixel_total : 8236 time to create 1 rle with old method : 0.013651371002197266 length of segment : 129 time for calcul the mask position with numpy : 0.0006148815155029297 nb_pixel_total : 6917 time to create 1 rle with old method : 0.010440349578857422 length of segment : 120 time for calcul the mask position with numpy : 0.002279996871948242 nb_pixel_total : 74149 time to create 1 rle with old method : 0.09086298942565918 length of segment : 270 time for calcul the mask position with numpy : 0.008571624755859375 nb_pixel_total : 318670 time to create 1 rle with new method : 0.011235475540161133 length of segment : 1002 time for calcul the mask position with numpy : 0.000370025634765625 nb_pixel_total : 5735 time to create 1 rle with old method : 0.0069921016693115234 length of segment : 90 time for calcul the mask position with numpy : 0.0006563663482666016 nb_pixel_total : 15463 time to create 1 rle with old method : 0.018352746963500977 length of segment : 218 time for calcul the mask position with numpy : 0.0019402503967285156 nb_pixel_total : 71249 time to create 1 rle with old method : 0.08437657356262207 length of segment : 322 time for calcul the mask position with numpy : 0.0038559436798095703 nb_pixel_total : 144406 time to create 1 rle with old method : 0.1665663719177246 length of segment : 599 time for calcul the mask position with numpy : 0.0016148090362548828 nb_pixel_total : 41200 time to create 1 rle with old method : 0.049503326416015625 length of segment : 354 time for calcul the mask position with numpy : 0.003914833068847656 nb_pixel_total : 123019 time to create 1 rle with old method : 0.13819599151611328 length of segment : 551 time for calcul the mask position with numpy : 0.00714421272277832 nb_pixel_total : 171405 time to create 1 rle with new method : 0.018752574920654297 length of segment : 758 time for calcul the mask position with numpy : 0.0007283687591552734 nb_pixel_total : 19206 time to create 1 rle with old method : 0.022891998291015625 length of segment : 434 time for calcul the mask position with numpy : 0.0011639595031738281 nb_pixel_total : 7636 time to create 1 rle with old method : 0.009169816970825195 length of segment : 209 time for calcul the mask position with numpy : 0.00040912628173828125 nb_pixel_total : 4992 time to create 1 rle with old method : 0.0063304901123046875 length of segment : 108 time for calcul the mask position with numpy : 0.00264739990234375 nb_pixel_total : 71109 time to create 1 rle with old method : 0.08221936225891113 length of segment : 660 time for calcul the mask position with numpy : 0.0008845329284667969 nb_pixel_total : 40932 time to create 1 rle with old method : 0.04932141304016113 length of segment : 262 time for calcul the mask position with numpy : 0.0004930496215820312 nb_pixel_total : 6091 time to create 1 rle with old method : 0.010198831558227539 length of segment : 130 time for calcul the mask position with numpy : 0.0035293102264404297 nb_pixel_total : 90306 time to create 1 rle with old method : 0.14049792289733887 length of segment : 380 time for calcul the mask position with numpy : 0.0032775402069091797 nb_pixel_total : 122771 time to create 1 rle with old method : 0.14045238494873047 length of segment : 346 time for calcul the mask position with numpy : 0.0001342296600341797 nb_pixel_total : 3446 time to create 1 rle with old method : 0.004263162612915039 length of segment : 91 time for calcul the mask position with numpy : 0.0007359981536865234 nb_pixel_total : 33567 time to create 1 rle with old method : 0.03869748115539551 length of segment : 157 time for calcul the mask position with numpy : 0.0002334117889404297 nb_pixel_total : 4201 time to create 1 rle with old method : 0.005040168762207031 length of segment : 88 time for calcul the mask position with numpy : 0.0003371238708496094 nb_pixel_total : 5677 time to create 1 rle with old method : 0.00672602653503418 length of segment : 139 time for calcul the mask position with numpy : 0.0020787715911865234 nb_pixel_total : 76183 time to create 1 rle with old method : 0.08542370796203613 length of segment : 584 time for calcul the mask position with numpy : 0.00039124488830566406 nb_pixel_total : 10612 time to create 1 rle with old method : 0.012804985046386719 length of segment : 97 time for calcul the mask position with numpy : 0.003622770309448242 nb_pixel_total : 159340 time to create 1 rle with new method : 0.005599498748779297 length of segment : 541 time for calcul the mask position with numpy : 0.0010602474212646484 nb_pixel_total : 25755 time to create 1 rle with old method : 0.029607057571411133 length of segment : 307 time for calcul the mask position with numpy : 0.0014982223510742188 nb_pixel_total : 43475 time to create 1 rle with old method : 0.050039052963256836 length of segment : 296 time for calcul the mask position with numpy : 0.0012431144714355469 nb_pixel_total : 52878 time to create 1 rle with old method : 0.08419680595397949 length of segment : 327 time for calcul the mask position with numpy : 0.002276897430419922 nb_pixel_total : 78942 time to create 1 rle with old method : 0.08896040916442871 length of segment : 284 time for calcul the mask position with numpy : 0.00037789344787597656 nb_pixel_total : 2481 time to create 1 rle with old method : 0.0032622814178466797 length of segment : 48 time for calcul the mask position with numpy : 0.0001232624053955078 nb_pixel_total : 2127 time to create 1 rle with old method : 0.00260162353515625 length of segment : 63 time for calcul the mask position with numpy : 0.003687620162963867 nb_pixel_total : 33987 time to create 1 rle with old method : 0.040163278579711914 length of segment : 357 time for calcul the mask position with numpy : 0.0001323223114013672 nb_pixel_total : 1993 time to create 1 rle with old method : 0.0026574134826660156 length of segment : 52 time for calcul the mask position with numpy : 0.0007395744323730469 nb_pixel_total : 31572 time to create 1 rle with old method : 0.03668212890625 length of segment : 158 time for calcul the mask position with numpy : 0.001390218734741211 nb_pixel_total : 40484 time to create 1 rle with old method : 0.048741817474365234 length of segment : 321 time for calcul the mask position with numpy : 0.0023369789123535156 nb_pixel_total : 66339 time to create 1 rle with old method : 0.15439176559448242 length of segment : 544 time for calcul the mask position with numpy : 0.00021576881408691406 nb_pixel_total : 2530 time to create 1 rle with old method : 0.0030629634857177734 length of segment : 95 time for calcul the mask position with numpy : 0.0014863014221191406 nb_pixel_total : 34168 time to create 1 rle with old method : 0.05285358428955078 length of segment : 291 time for calcul the mask position with numpy : 0.00033020973205566406 nb_pixel_total : 12443 time to create 1 rle with old method : 0.014515399932861328 length of segment : 228 time for calcul the mask position with numpy : 0.0001347064971923828 nb_pixel_total : 4489 time to create 1 rle with old method : 0.005415201187133789 length of segment : 107 time for calcul the mask position with numpy : 0.00043702125549316406 nb_pixel_total : 13919 time to create 1 rle with old method : 0.016584396362304688 length of segment : 123 time for calcul the mask position with numpy : 0.0002636909484863281 nb_pixel_total : 4194 time to create 1 rle with old method : 0.005093812942504883 length of segment : 123 time for calcul the mask position with numpy : 0.0018582344055175781 nb_pixel_total : 87572 time to create 1 rle with old method : 0.10092306137084961 length of segment : 331 time for calcul the mask position with numpy : 0.00023794174194335938 nb_pixel_total : 2694 time to create 1 rle with old method : 0.003590106964111328 length of segment : 59 time for calcul the mask position with numpy : 0.001495361328125 nb_pixel_total : 56245 time to create 1 rle with old method : 0.0662229061126709 length of segment : 231 time for calcul the mask position with numpy : 0.0004987716674804688 nb_pixel_total : 10707 time to create 1 rle with old method : 0.012806177139282227 length of segment : 159 time for calcul the mask position with numpy : 0.0014934539794921875 nb_pixel_total : 32346 time to create 1 rle with old method : 0.03690981864929199 length of segment : 301 time for calcul the mask position with numpy : 0.0017502307891845703 nb_pixel_total : 71372 time to create 1 rle with old method : 0.08030056953430176 length of segment : 369 time for calcul the mask position with numpy : 0.00020503997802734375 nb_pixel_total : 1578 time to create 1 rle with old method : 0.0019314289093017578 length of segment : 66 time for calcul the mask position with numpy : 0.0020513534545898438 nb_pixel_total : 85920 time to create 1 rle with old method : 0.12048935890197754 length of segment : 651 time for calcul the mask position with numpy : 0.001508951187133789 nb_pixel_total : 60875 time to create 1 rle with old method : 0.0893402099609375 length of segment : 335 time for calcul the mask position with numpy : 0.0012385845184326172 nb_pixel_total : 54007 time to create 1 rle with old method : 0.06243538856506348 length of segment : 307 time for calcul the mask position with numpy : 0.00027561187744140625 nb_pixel_total : 2419 time to create 1 rle with old method : 0.0029299259185791016 length of segment : 92 time for calcul the mask position with numpy : 0.0007939338684082031 nb_pixel_total : 23275 time to create 1 rle with old method : 0.02718353271484375 length of segment : 243 time for calcul the mask position with numpy : 0.0011506080627441406 nb_pixel_total : 36675 time to create 1 rle with old method : 0.043885231018066406 length of segment : 204 time for calcul the mask position with numpy : 0.00018262863159179688 nb_pixel_total : 2673 time to create 1 rle with old method : 0.0032758712768554688 length of segment : 81 time for calcul the mask position with numpy : 0.00018596649169921875 nb_pixel_total : 2096 time to create 1 rle with old method : 0.002629518508911133 length of segment : 78 time for calcul the mask position with numpy : 0.0016856193542480469 nb_pixel_total : 74979 time to create 1 rle with old method : 0.08551621437072754 length of segment : 275 time for calcul the mask position with numpy : 0.00019860267639160156 nb_pixel_total : 1833 time to create 1 rle with old method : 0.0022552013397216797 length of segment : 81 time for calcul the mask position with numpy : 0.0076673030853271484 nb_pixel_total : 248174 time to create 1 rle with new method : 0.015816450119018555 length of segment : 834 time for calcul the mask position with numpy : 0.0031213760375976562 nb_pixel_total : 110855 time to create 1 rle with old method : 0.14038801193237305 length of segment : 610 time for calcul the mask position with numpy : 0.0008826255798339844 nb_pixel_total : 27476 time to create 1 rle with old method : 0.03151512145996094 length of segment : 223 time for calcul the mask position with numpy : 0.0010371208190917969 nb_pixel_total : 21903 time to create 1 rle with old method : 0.026128292083740234 length of segment : 232 time for calcul the mask position with numpy : 0.004276752471923828 nb_pixel_total : 169504 time to create 1 rle with new method : 0.009495019912719727 length of segment : 788 time for calcul the mask position with numpy : 0.0009229183197021484 nb_pixel_total : 27576 time to create 1 rle with old method : 0.03189682960510254 length of segment : 349 time for calcul the mask position with numpy : 0.0015702247619628906 nb_pixel_total : 57135 time to create 1 rle with old method : 0.0673065185546875 length of segment : 278 time for calcul the mask position with numpy : 0.006030082702636719 nb_pixel_total : 209006 time to create 1 rle with new method : 0.00978398323059082 length of segment : 1401 time for calcul the mask position with numpy : 0.00521087646484375 nb_pixel_total : 225396 time to create 1 rle with new method : 0.008065938949584961 length of segment : 671 time for calcul the mask position with numpy : 0.0001838207244873047 nb_pixel_total : 1615 time to create 1 rle with old method : 0.002003192901611328 length of segment : 71 time for calcul the mask position with numpy : 0.00118255615234375 nb_pixel_total : 45125 time to create 1 rle with old method : 0.05208539962768555 length of segment : 244 time for calcul the mask position with numpy : 0.001262664794921875 nb_pixel_total : 51808 time to create 1 rle with old method : 0.059175968170166016 length of segment : 288 time for calcul the mask position with numpy : 0.0002167224884033203 nb_pixel_total : 4382 time to create 1 rle with old method : 0.0052220821380615234 length of segment : 88 time for calcul the mask position with numpy : 0.005091428756713867 nb_pixel_total : 219211 time to create 1 rle with new method : 0.007819652557373047 length of segment : 468 time for calcul the mask position with numpy : 0.0002658367156982422 nb_pixel_total : 4604 time to create 1 rle with old method : 0.005779266357421875 length of segment : 77 time for calcul the mask position with numpy : 0.00021529197692871094 nb_pixel_total : 3281 time to create 1 rle with old method : 0.004322052001953125 length of segment : 63 time for calcul the mask position with numpy : 0.00024819374084472656 nb_pixel_total : 4463 time to create 1 rle with old method : 0.0077397823333740234 length of segment : 49 time for calcul the mask position with numpy : 0.003251791000366211 nb_pixel_total : 76040 time to create 1 rle with old method : 0.0996699333190918 length of segment : 413 time for calcul the mask position with numpy : 0.0006005764007568359 nb_pixel_total : 15020 time to create 1 rle with old method : 0.02202892303466797 length of segment : 167 time for calcul the mask position with numpy : 0.0004439353942871094 nb_pixel_total : 13507 time to create 1 rle with old method : 0.03876495361328125 length of segment : 109 time for calcul the mask position with numpy : 0.006545305252075195 nb_pixel_total : 181044 time to create 1 rle with new method : 0.00924825668334961 length of segment : 479 time for calcul the mask position with numpy : 0.006693124771118164 nb_pixel_total : 161569 time to create 1 rle with new method : 0.013421058654785156 length of segment : 471 time for calcul the mask position with numpy : 0.0009453296661376953 nb_pixel_total : 29617 time to create 1 rle with old method : 0.03501272201538086 length of segment : 131 time for calcul the mask position with numpy : 0.0005772113800048828 nb_pixel_total : 18977 time to create 1 rle with old method : 0.023284196853637695 length of segment : 117 time for calcul the mask position with numpy : 0.00030994415283203125 nb_pixel_total : 5694 time to create 1 rle with old method : 0.006906032562255859 length of segment : 90 time for calcul the mask position with numpy : 0.0007212162017822266 nb_pixel_total : 15295 time to create 1 rle with old method : 0.018791913986206055 length of segment : 200 time for calcul the mask position with numpy : 0.0025115013122558594 nb_pixel_total : 83267 time to create 1 rle with old method : 0.09478068351745605 length of segment : 424 time for calcul the mask position with numpy : 0.00035190582275390625 nb_pixel_total : 5574 time to create 1 rle with old method : 0.007131099700927734 length of segment : 80 time spent for convertir_results : 21.59487247467041 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 132 chid ids of type : 3663 Number RLEs to save : 58623 save missing photos in datou_result : time spend for datou_step_exec : 71.16707944869995 time spend to save output : 4.286142587661743 total time spend for step 1 : 75.4532220363617 step2:crop_condition Fri Mar 14 11:23:45 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 132 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 16 About to insert : list_path_to_insert length 16 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 43 About to insert : list_path_to_insert length 43 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1344106575, 1344106568, 1344106564, 1344106325, 1344106321, 1344106314, 1344106308, 1344106300, 1344106225, 1344106191] Looping around the photos to save general results len do output : 82 /-3711777839Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777843Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777862Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777866Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777860Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777863Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777859Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777881Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777882Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777900Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777909Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777904Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777926Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777938Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777953Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777957Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777916Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777919Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777921Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777943Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777950Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777951Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777948Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777858Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777870Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777897Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777923Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777966Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777841Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777840Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777838Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777849Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777848Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777855Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777853Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777864Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777873Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777880Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777878Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777857Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777871Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777876Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777874Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777879Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777877Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777889Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777888Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777891Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777895Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777886Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777885Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777893Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777892Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777908Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777912Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777903Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777907Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777915Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777918Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777920Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777930Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777927Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777928Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777932Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777925Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777934Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777931Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777946Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777960Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777965Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777961Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777902Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777962Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777959Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777958Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777867Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777887Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777898Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777935Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777942Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777963Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3711777967Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106575', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106568', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106564', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106325', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106321', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106314', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106308', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106300', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106225', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106191', None, None, None, None, None, '2668799') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 256 time used for this insertion : 0.07432961463928223 save_final save missing photos in datou_result : time spend for datou_step_exec : 43.33928823471069 time spend to save output : 0.4383978843688965 total time spend for step 2 : 43.77768611907959 step3:thcl Fri Mar 14 11:24:28 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.009127378463745117 time to convert the images to numpy array : 0.0021104812622070312 time to import caffe and check if the image exist : 0.016282081604003906 time to convert the images to numpy array : 0.32639622688293457 time to import caffe and check if the image exist : 0.0079193115234375 time to convert the images to numpy array : 0.3629744052886963 time to import caffe and check if the image exist : 0.022612810134887695 time to convert the images to numpy array : 0.35132932662963867 time to import caffe and check if the image exist : 0.017976760864257812 time to convert the images to numpy array : 0.35917043685913086 time to import caffe and check if the image exist : 0.015193939208984375 time to convert the images to numpy array : 0.36298131942749023 time to import caffe and check if the image exist : 0.016168832778930664 time to convert the images to numpy array : 0.3718600273132324 time to import caffe and check if the image exist : 0.02259540557861328 time to convert the images to numpy array : 0.3783857822418213 time to import caffe and check if the image exist : 0.019927263259887695 time to convert the images to numpy array : 0.38131093978881836 time to import caffe and check if the image exist : 0.019332408905029297 time to convert the images to numpy array : 0.7168922424316406 total time to convert the images to numpy array : 0.9892022609710693 list photo_ids error: [] list photo_ids correct : [-3711777967, -3711777889, -3711777888, -3711777891, -3711777895, -3711777886, -3711777885, -3711777893, -3711777892, -3711777908, -3711777839, -3711777843, -3711777862, -3711777866, -3711777860, -3711777863, -3711777859, -3711777881, -3711777882, -3711777873, -3711777880, -3711777878, -3711777857, -3711777871, -3711777876, -3711777874, -3711777879, -3711777877, -3711777912, -3711777903, -3711777907, -3711777915, -3711777918, -3711777920, -3711777930, -3711777927, -3711777928, -3711777962, -3711777959, -3711777958, -3711777867, -3711777887, -3711777898, -3711777935, -3711777942, -3711777963, -3711777900, -3711777909, -3711777904, -3711777926, -3711777938, -3711777953, -3711777957, -3711777916, -3711777919, -3711777921, -3711777943, -3711777950, -3711777951, -3711777948, -3711777858, -3711777870, -3711777897, -3711777923, -3711777932, -3711777925, -3711777934, -3711777931, -3711777946, -3711777960, -3711777965, -3711777961, -3711777902, -3711777966, -3711777841, -3711777840, -3711777838, -3711777849, -3711777848, -3711777855, -3711777853, -3711777864] number of photos to traite : 82 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 6717 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3270 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.2279353141784668 time used to do the prediction : 0.20704865455627441 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3268 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3100 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.1703686714172363 time used to do the prediction : 0.15332984924316406 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 1.811981201171875e-05 save missing photos in datou_result : time spend for datou_step_exec : 10.516182661056519 time spend to save output : 0.0006899833679199219 total time spend for step 3 : 10.516872644424438 step4:merge_mask_thcl_custom Fri Mar 14 11:24:39 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 132 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1344106575, 1344106568, 1344106564, 1344106325, 1344106321, 1344106314, 1344106308, 1344106300, 1344106225, 1344106191] Looping around the photos to save general results len do output : 10 /1344106575Didn't retrieve data .Didn't retrieve data . /1344106568Didn't retrieve data .Didn't retrieve data . /1344106564Didn't retrieve data .Didn't retrieve data . /1344106325Didn't retrieve data .Didn't retrieve data . /1344106321Didn't retrieve data .Didn't retrieve data . /1344106314Didn't retrieve data .Didn't retrieve data . /1344106308Didn't retrieve data .Didn't retrieve data . /1344106300Didn't retrieve data .Didn't retrieve data . /1344106225Didn't retrieve data .Didn't retrieve data . /1344106191Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106575', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106568', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106564', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106325', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106321', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106314', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106308', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106300', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106225', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106191', None, None, None, None, None, '2668799') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013034343719482422 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.10485649108886719 time spend to save output : 0.01367640495300293 total time spend for step 4 : 0.11853289604187012 step5:rle_unique_nms_with_priority Fri Mar 14 11:24:39 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 6 nb_hashtags : 3 time to prepare the origin masks : 4.043144941329956 time for calcul the mask position with numpy : 0.03516960144042969 nb_pixel_total : 2203041 time to create 1 rle with new method : 0.29446840286254883 time for calcul the mask position with numpy : 0.023320674896240234 nb_pixel_total : 40723 time to create 1 rle with old method : 0.06506085395812988 time for calcul the mask position with numpy : 0.2210390567779541 nb_pixel_total : 4579731 time to create 1 rle with new method : 0.3715677261352539 time for calcul the mask position with numpy : 0.022090435028076172 nb_pixel_total : 5064 time to create 1 rle with old method : 0.0060846805572509766 time for calcul the mask position with numpy : 0.02337646484375 nb_pixel_total : 109735 time to create 1 rle with old method : 0.13199734687805176 time for calcul the mask position with numpy : 0.027063846588134766 nb_pixel_total : 19546 time to create 1 rle with old method : 0.031095027923583984 time for calcul the mask position with numpy : 0.023708581924438477 nb_pixel_total : 92400 time to create 1 rle with old method : 0.11179423332214355 create new chi : 1.4430830478668213 time to delete rle : 0.01527714729309082 batch 1 Loaded 7 chid ids of type : 3726 Number RLEs to save : 12851 TO DO : save crop sub photo not yet done ! save time : 2.6706254482269287 nb_obj : 9 nb_hashtags : 3 time to prepare the origin masks : 5.5013792514801025 time for calcul the mask position with numpy : 0.04407930374145508 nb_pixel_total : 2566224 time to create 1 rle with new method : 0.3419983386993408 time for calcul the mask position with numpy : 0.02826547622680664 nb_pixel_total : 377120 time to create 1 rle with new method : 0.35523033142089844 time for calcul the mask position with numpy : 0.021930217742919922 nb_pixel_total : 121716 time to create 1 rle with old method : 0.16077804565429688 time for calcul the mask position with numpy : 0.02203845977783203 nb_pixel_total : 169517 time to create 1 rle with new method : 0.2965219020843506 time for calcul the mask position with numpy : 0.023117780685424805 nb_pixel_total : 60038 time to create 1 rle with old method : 0.07746171951293945 time for calcul the mask position with numpy : 0.022127151489257812 nb_pixel_total : 77456 time to create 1 rle with old method : 0.08493328094482422 time for calcul the mask position with numpy : 0.024504423141479492 nb_pixel_total : 347123 time to create 1 rle with new method : 0.3882262706756592 time for calcul the mask position with numpy : 0.021692514419555664 nb_pixel_total : 199020 time to create 1 rle with new method : 0.291231632232666 time for calcul the mask position with numpy : 0.025789976119995117 nb_pixel_total : 312945 time to create 1 rle with new method : 0.3615913391113281 time for calcul the mask position with numpy : 0.16655874252319336 nb_pixel_total : 2819081 time to create 1 rle with new method : 0.40352535247802734 create new chi : 3.3366405963897705 time to delete rle : 0.0016682147979736328 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 17412 TO DO : save crop sub photo not yet done ! save time : 1.4262819290161133 nb_obj : 5 nb_hashtags : 3 time to prepare the origin masks : 4.147344350814819 time for calcul the mask position with numpy : 0.12688827514648438 nb_pixel_total : 3515946 time to create 1 rle with new method : 0.3310527801513672 time for calcul the mask position with numpy : 0.022553682327270508 nb_pixel_total : 48275 time to create 1 rle with old method : 0.05923962593078613 time for calcul the mask position with numpy : 0.02333521842956543 nb_pixel_total : 82731 time to create 1 rle with old method : 0.10695385932922363 time for calcul the mask position with numpy : 0.03802800178527832 nb_pixel_total : 317815 time to create 1 rle with new method : 0.5443665981292725 time for calcul the mask position with numpy : 0.07096076011657715 nb_pixel_total : 3020754 time to create 1 rle with new method : 0.5020110607147217 time for calcul the mask position with numpy : 0.03308463096618652 nb_pixel_total : 64719 time to create 1 rle with old method : 0.07229185104370117 create new chi : 2.0151216983795166 time to delete rle : 0.0021905899047851562 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 13645 TO DO : save crop sub photo not yet done ! save time : 1.9221136569976807 nb_obj : 23 nb_hashtags : 7 time to prepare the origin masks : 10.12344241142273 time for calcul the mask position with numpy : 0.19765710830688477 nb_pixel_total : 5673689 time to create 1 rle with new method : 1.7959082126617432 time for calcul the mask position with numpy : 0.03192853927612305 nb_pixel_total : 33692 time to create 1 rle with old method : 0.03910040855407715 time for calcul the mask position with numpy : 0.03652834892272949 nb_pixel_total : 8992 time to create 1 rle with old method : 0.012353658676147461 time for calcul the mask position with numpy : 0.0266416072845459 nb_pixel_total : 47895 time to create 1 rle with old method : 0.06057453155517578 time for calcul the mask position with numpy : 0.025777578353881836 nb_pixel_total : 109006 time to create 1 rle with old method : 0.11842703819274902 time for calcul the mask position with numpy : 0.024497270584106445 nb_pixel_total : 2988 time to create 1 rle with old method : 0.003847360610961914 time for calcul the mask position with numpy : 0.028568267822265625 nb_pixel_total : 37887 time to create 1 rle with old method : 0.04136300086975098 time for calcul the mask position with numpy : 0.02778482437133789 nb_pixel_total : 75710 time to create 1 rle with old method : 0.08134007453918457 time for calcul the mask position with numpy : 0.026829004287719727 nb_pixel_total : 105945 time to create 1 rle with old method : 0.11573576927185059 time for calcul the mask position with numpy : 0.021762371063232422 nb_pixel_total : 18964 time to create 1 rle with old method : 0.021199464797973633 time for calcul the mask position with numpy : 0.026676416397094727 nb_pixel_total : 74418 time to create 1 rle with old method : 0.08686280250549316 time for calcul the mask position with numpy : 0.033272743225097656 nb_pixel_total : 44203 time to create 1 rle with old method : 0.049062490463256836 time for calcul the mask position with numpy : 0.03204226493835449 nb_pixel_total : 35781 time to create 1 rle with old method : 0.037950754165649414 time for calcul the mask position with numpy : 0.03351283073425293 nb_pixel_total : 3232 time to create 1 rle with old method : 0.007806539535522461 time for calcul the mask position with numpy : 0.03366208076477051 nb_pixel_total : 116225 time to create 1 rle with old method : 0.12564587593078613 time for calcul the mask position with numpy : 0.03296399116516113 nb_pixel_total : 1540 time to create 1 rle with old method : 0.0017695426940917969 time for calcul the mask position with numpy : 0.032865047454833984 nb_pixel_total : 16832 time to create 1 rle with old method : 0.01796889305114746 time for calcul the mask position with numpy : 0.03272533416748047 nb_pixel_total : 110293 time to create 1 rle with old method : 0.11791729927062988 time for calcul the mask position with numpy : 0.03653430938720703 nb_pixel_total : 7502 time to create 1 rle with old method : 0.008567571640014648 time for calcul the mask position with numpy : 0.032288312911987305 nb_pixel_total : 470 time to create 1 rle with old method : 0.0006427764892578125 time for calcul the mask position with numpy : 0.03673839569091797 nb_pixel_total : 32894 time to create 1 rle with old method : 0.03476238250732422 time for calcul the mask position with numpy : 0.0321354866027832 nb_pixel_total : 8149 time to create 1 rle with old method : 0.00903177261352539 time for calcul the mask position with numpy : 0.03505277633666992 nb_pixel_total : 321685 time to create 1 rle with new method : 0.2827913761138916 time for calcul the mask position with numpy : 0.0348052978515625 nb_pixel_total : 162248 time to create 1 rle with new method : 0.2946491241455078 create new chi : 4.355820178985596 time to delete rle : 0.00154876708984375 batch 1 Loaded 24 chid ids of type : 3726 Number RLEs to save : 15374 TO DO : save crop sub photo not yet done ! save time : 0.9883897304534912 nb_obj : 19 nb_hashtags : 5 time to prepare the origin masks : 8.028269290924072 time for calcul the mask position with numpy : 0.24256467819213867 nb_pixel_total : 5719588 time to create 1 rle with new method : 0.3173806667327881 time for calcul the mask position with numpy : 0.03958916664123535 nb_pixel_total : 110550 time to create 1 rle with old method : 0.12286901473999023 time for calcul the mask position with numpy : 0.03132462501525879 nb_pixel_total : 7624 time to create 1 rle with old method : 0.008737325668334961 time for calcul the mask position with numpy : 0.021281003952026367 nb_pixel_total : 18894 time to create 1 rle with old method : 0.021631479263305664 time for calcul the mask position with numpy : 0.02246999740600586 nb_pixel_total : 3435 time to create 1 rle with old method : 0.003966569900512695 time for calcul the mask position with numpy : 0.021411895751953125 nb_pixel_total : 90185 time to create 1 rle with old method : 0.10265612602233887 time for calcul the mask position with numpy : 0.02330946922302246 nb_pixel_total : 74041 time to create 1 rle with old method : 0.0826723575592041 time for calcul the mask position with numpy : 0.02143549919128418 nb_pixel_total : 6086 time to create 1 rle with old method : 0.006818056106567383 time for calcul the mask position with numpy : 0.021671056747436523 nb_pixel_total : 15403 time to create 1 rle with old method : 0.017157316207885742 time for calcul the mask position with numpy : 0.021662473678588867 nb_pixel_total : 70643 time to create 1 rle with old method : 0.08623528480529785 time for calcul the mask position with numpy : 0.022121191024780273 nb_pixel_total : 4953 time to create 1 rle with old method : 0.0058019161224365234 time for calcul the mask position with numpy : 0.025077342987060547 nb_pixel_total : 71041 time to create 1 rle with old method : 0.08304667472839355 time for calcul the mask position with numpy : 0.02979874610900879 nb_pixel_total : 170901 time to create 1 rle with new method : 0.3848438262939453 time for calcul the mask position with numpy : 0.0338284969329834 nb_pixel_total : 6891 time to create 1 rle with old method : 0.00777888298034668 time for calcul the mask position with numpy : 0.034349918365478516 nb_pixel_total : 5717 time to create 1 rle with old method : 0.006645679473876953 time for calcul the mask position with numpy : 0.033625125885009766 nb_pixel_total : 48548 time to create 1 rle with old method : 0.0588078498840332 time for calcul the mask position with numpy : 0.03922605514526367 nb_pixel_total : 318470 time to create 1 rle with new method : 0.36690616607666016 time for calcul the mask position with numpy : 0.03462624549865723 nb_pixel_total : 40824 time to create 1 rle with old method : 0.051275014877319336 time for calcul the mask position with numpy : 0.044037580490112305 nb_pixel_total : 144115 time to create 1 rle with old method : 0.16545486450195312 time for calcul the mask position with numpy : 0.036791324615478516 nb_pixel_total : 122331 time to create 1 rle with old method : 0.16351914405822754 create new chi : 2.9516422748565674 time to delete rle : 0.0015552043914794922 batch 1 Loaded 20 chid ids of type : 3726 Number RLEs to save : 15465 TO DO : save crop sub photo not yet done ! save time : 1.1133146286010742 nb_obj : 14 nb_hashtags : 6 time to prepare the origin masks : 7.915351390838623 time for calcul the mask position with numpy : 0.3675103187561035 nb_pixel_total : 6543652 time to create 1 rle with new method : 0.3236064910888672 time for calcul the mask position with numpy : 0.025498628616333008 nb_pixel_total : 33428 time to create 1 rle with old method : 0.03988981246948242 time for calcul the mask position with numpy : 0.03043365478515625 nb_pixel_total : 44505 time to create 1 rle with old method : 0.05431723594665527 time for calcul the mask position with numpy : 0.02248978614807129 nb_pixel_total : 1979 time to create 1 rle with old method : 0.002370595932006836 time for calcul the mask position with numpy : 0.02227926254272461 nb_pixel_total : 10560 time to create 1 rle with old method : 0.014076471328735352 time for calcul the mask position with numpy : 0.023048877716064453 nb_pixel_total : 2124 time to create 1 rle with old method : 0.002510547637939453 time for calcul the mask position with numpy : 0.02255988121032715 nb_pixel_total : 5675 time to create 1 rle with old method : 0.006755828857421875 time for calcul the mask position with numpy : 0.02214837074279785 nb_pixel_total : 75891 time to create 1 rle with old method : 0.08820772171020508 time for calcul the mask position with numpy : 0.021111726760864258 nb_pixel_total : 2430 time to create 1 rle with old method : 0.0027844905853271484 time for calcul the mask position with numpy : 0.026205062866210938 nb_pixel_total : 27160 time to create 1 rle with old method : 0.03121161460876465 time for calcul the mask position with numpy : 0.020740747451782227 nb_pixel_total : 25698 time to create 1 rle with old method : 0.028303861618041992 time for calcul the mask position with numpy : 0.021950721740722656 nb_pixel_total : 160724 time to create 1 rle with new method : 0.29459547996520996 time for calcul the mask position with numpy : 0.02120375633239746 nb_pixel_total : 4188 time to create 1 rle with old method : 0.0049474239349365234 time for calcul the mask position with numpy : 0.023131608963012695 nb_pixel_total : 78704 time to create 1 rle with old method : 0.08701705932617188 time for calcul the mask position with numpy : 0.021089553833007812 nb_pixel_total : 33522 time to create 1 rle with old method : 0.0394132137298584 create new chi : 1.770076036453247 time to delete rle : 0.0010044574737548828 batch 1 Loaded 15 chid ids of type : 3726 Number RLEs to save : 8502 TO DO : save crop sub photo not yet done ! save time : 0.7007598876953125 nb_obj : 11 nb_hashtags : 5 time to prepare the origin masks : 7.139681339263916 time for calcul the mask position with numpy : 0.30063867568969727 nb_pixel_total : 6759444 time to create 1 rle with new method : 0.2962768077850342 time for calcul the mask position with numpy : 0.02786564826965332 nb_pixel_total : 31394 time to create 1 rle with old method : 0.03685259819030762 time for calcul the mask position with numpy : 0.027639150619506836 nb_pixel_total : 60325 time to create 1 rle with old method : 0.07153868675231934 time for calcul the mask position with numpy : 0.02458810806274414 nb_pixel_total : 12388 time to create 1 rle with old method : 0.015069246292114258 time for calcul the mask position with numpy : 0.024306058883666992 nb_pixel_total : 40376 time to create 1 rle with old method : 0.04843282699584961 time for calcul the mask position with numpy : 0.026140689849853516 nb_pixel_total : 2657 time to create 1 rle with old method : 0.003290891647338867 time for calcul the mask position with numpy : 0.02391338348388672 nb_pixel_total : 13903 time to create 1 rle with old method : 0.0167849063873291 time for calcul the mask position with numpy : 0.02555537223815918 nb_pixel_total : 4184 time to create 1 rle with old method : 0.005079507827758789 time for calcul the mask position with numpy : 0.02370429039001465 nb_pixel_total : 2517 time to create 1 rle with old method : 0.002991914749145508 time for calcul the mask position with numpy : 0.022925615310668945 nb_pixel_total : 4470 time to create 1 rle with old method : 0.005076885223388672 time for calcul the mask position with numpy : 0.023745298385620117 nb_pixel_total : 87177 time to create 1 rle with old method : 0.12501072883605957 time for calcul the mask position with numpy : 0.022193193435668945 nb_pixel_total : 31405 time to create 1 rle with old method : 0.037126779556274414 create new chi : 1.2755482196807861 time to delete rle : 0.0008218288421630859 batch 1 Loaded 12 chid ids of type : 3726 Number RLEs to save : 6644 TO DO : save crop sub photo not yet done ! save time : 0.8597588539123535 nb_obj : 13 nb_hashtags : 6 time to prepare the origin masks : 7.78459095954895 time for calcul the mask position with numpy : 0.5781488418579102 nb_pixel_total : 6611288 time to create 1 rle with new method : 0.6455464363098145 time for calcul the mask position with numpy : 0.025639057159423828 nb_pixel_total : 2091 time to create 1 rle with old method : 0.002549409866333008 time for calcul the mask position with numpy : 0.028752565383911133 nb_pixel_total : 60803 time to create 1 rle with old method : 0.06995654106140137 time for calcul the mask position with numpy : 0.028007030487060547 nb_pixel_total : 32226 time to create 1 rle with old method : 0.0371701717376709 time for calcul the mask position with numpy : 0.02534961700439453 nb_pixel_total : 2663 time to create 1 rle with old method : 0.0030651092529296875 time for calcul the mask position with numpy : 0.025812864303588867 nb_pixel_total : 23238 time to create 1 rle with old method : 0.02770233154296875 time for calcul the mask position with numpy : 0.025738239288330078 nb_pixel_total : 2408 time to create 1 rle with old method : 0.0029289722442626953 time for calcul the mask position with numpy : 0.027244091033935547 nb_pixel_total : 85617 time to create 1 rle with old method : 0.10062551498413086 time for calcul the mask position with numpy : 0.02674269676208496 nb_pixel_total : 10677 time to create 1 rle with old method : 0.014486074447631836 time for calcul the mask position with numpy : 0.03040003776550293 nb_pixel_total : 1567 time to create 1 rle with old method : 0.001844644546508789 time for calcul the mask position with numpy : 0.03038763999938965 nb_pixel_total : 71089 time to create 1 rle with old method : 0.07913756370544434 time for calcul the mask position with numpy : 0.029194116592407227 nb_pixel_total : 36588 time to create 1 rle with old method : 0.04916858673095703 time for calcul the mask position with numpy : 0.032726287841796875 nb_pixel_total : 56055 time to create 1 rle with old method : 0.07663631439208984 time for calcul the mask position with numpy : 0.034657955169677734 nb_pixel_total : 53930 time to create 1 rle with old method : 0.0781242847442627 create new chi : 2.1859660148620605 time to delete rle : 0.0014846324920654297 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 8366 TO DO : save crop sub photo not yet done ! save time : 0.5142252445220947 nb_obj : 14 nb_hashtags : 4 time to prepare the origin masks : 8.116467237472534 time for calcul the mask position with numpy : 0.26650547981262207 nb_pixel_total : 5849922 time to create 1 rle with new method : 0.46038031578063965 time for calcul the mask position with numpy : 0.02876591682434082 nb_pixel_total : 56978 time to create 1 rle with old method : 0.06425738334655762 time for calcul the mask position with numpy : 0.029016494750976562 nb_pixel_total : 74864 time to create 1 rle with old method : 0.11721038818359375 time for calcul the mask position with numpy : 0.028565406799316406 nb_pixel_total : 198107 time to create 1 rle with new method : 0.407487154006958 time for calcul the mask position with numpy : 0.027333736419677734 nb_pixel_total : 51667 time to create 1 rle with old method : 0.059120893478393555 time for calcul the mask position with numpy : 0.028629302978515625 nb_pixel_total : 1601 time to create 1 rle with old method : 0.00197601318359375 time for calcul the mask position with numpy : 0.03215217590332031 nb_pixel_total : 165822 time to create 1 rle with new method : 0.4719581604003906 time for calcul the mask position with numpy : 0.04407191276550293 nb_pixel_total : 45025 time to create 1 rle with old method : 0.05103158950805664 time for calcul the mask position with numpy : 0.04195451736450195 nb_pixel_total : 27309 time to create 1 rle with old method : 0.03494143486022949 time for calcul the mask position with numpy : 0.04427289962768555 nb_pixel_total : 169185 time to create 1 rle with new method : 0.42531490325927734 time for calcul the mask position with numpy : 0.038979291915893555 nb_pixel_total : 27380 time to create 1 rle with old method : 0.03260445594787598 time for calcul the mask position with numpy : 0.04172801971435547 nb_pixel_total : 21804 time to create 1 rle with old method : 0.025501012802124023 time for calcul the mask position with numpy : 0.04417538642883301 nb_pixel_total : 110622 time to create 1 rle with old method : 0.13906431198120117 time for calcul the mask position with numpy : 0.03900289535522461 nb_pixel_total : 248125 time to create 1 rle with new method : 0.45029234886169434 time for calcul the mask position with numpy : 0.027528047561645508 nb_pixel_total : 1829 time to create 1 rle with old method : 0.0022749900817871094 create new chi : 3.6531171798706055 time to delete rle : 0.001722574234008789 batch 1 Loaded 15 chid ids of type : 3726 Number RLEs to save : 14082 TO DO : save crop sub photo not yet done ! save time : 0.817434549331665 nb_obj : 15 nb_hashtags : 6 time to prepare the origin masks : 9.016449451446533 time for calcul the mask position with numpy : 0.910754919052124 nb_pixel_total : 6295021 time to create 1 rle with new method : 1.7844011783599854 time for calcul the mask position with numpy : 0.030351877212524414 nb_pixel_total : 4555 time to create 1 rle with old method : 0.005811452865600586 time for calcul the mask position with numpy : 0.02960371971130371 nb_pixel_total : 15196 time to create 1 rle with old method : 0.02039337158203125 time for calcul the mask position with numpy : 0.03731179237365723 nb_pixel_total : 43 time to create 1 rle with old method : 0.0002243518829345703 time for calcul the mask position with numpy : 0.04121112823486328 nb_pixel_total : 159123 time to create 1 rle with new method : 1.2005999088287354 time for calcul the mask position with numpy : 0.03199434280395508 nb_pixel_total : 29270 time to create 1 rle with old method : 0.03522610664367676 time for calcul the mask position with numpy : 0.039611101150512695 nb_pixel_total : 180411 time to create 1 rle with new method : 0.7348721027374268 time for calcul the mask position with numpy : 0.030029773712158203 nb_pixel_total : 82902 time to create 1 rle with old method : 0.09801006317138672 time for calcul the mask position with numpy : 0.03283882141113281 nb_pixel_total : 5663 time to create 1 rle with old method : 0.006774187088012695 time for calcul the mask position with numpy : 0.03256702423095703 nb_pixel_total : 13395 time to create 1 rle with old method : 0.015609502792358398 time for calcul the mask position with numpy : 0.033675193786621094 nb_pixel_total : 14974 time to create 1 rle with old method : 0.01775813102722168 time for calcul the mask position with numpy : 0.0326077938079834 nb_pixel_total : 218468 time to create 1 rle with new method : 0.83510422706604 time for calcul the mask position with numpy : 0.02283167839050293 nb_pixel_total : 4451 time to create 1 rle with old method : 0.0051441192626953125 time for calcul the mask position with numpy : 0.022945880889892578 nb_pixel_total : 3256 time to create 1 rle with old method : 0.003799915313720703 time for calcul the mask position with numpy : 0.022538185119628906 nb_pixel_total : 4585 time to create 1 rle with old method : 0.09233212471008301 time for calcul the mask position with numpy : 0.024654865264892578 nb_pixel_total : 18927 time to create 1 rle with old method : 0.029142141342163086 create new chi : 6.4114601612091064 time to delete rle : 0.0009446144104003906 batch 1 Loaded 16 chid ids of type : 3726 Number RLEs to save : 8031 TO DO : save crop sub photo not yet done ! save time : 0.49515342712402344 map_output_result : {1344106575: (0.0, 'Should be the crop_list due to order', 0), 1344106568: (0.0, 'Should be the crop_list due to order', 0), 1344106564: (0.0, 'Should be the crop_list due to order', 0), 1344106325: (0.0, 'Should be the crop_list due to order', 0), 1344106321: (0.0, 'Should be the crop_list due to order', 0), 1344106314: (0.0, 'Should be the crop_list due to order', 0), 1344106308: (0.0, 'Should be the crop_list due to order', 0), 1344106300: (0.0, 'Should be the crop_list due to order', 0), 1344106225: (0.0, 'Should be the crop_list due to order', 0), 1344106191: (0.0, 'Should be the crop_list due to order', 0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1344106575, 1344106568, 1344106564, 1344106325, 1344106321, 1344106314, 1344106308, 1344106300, 1344106225, 1344106191] Looping around the photos to save general results len do output : 10 /1344106575.Didn't retrieve data . /1344106568.Didn't retrieve data . /1344106564.Didn't retrieve data . /1344106325.Didn't retrieve data . /1344106321.Didn't retrieve data . /1344106314.Didn't retrieve data . /1344106308.Didn't retrieve data . /1344106300.Didn't retrieve data . /1344106225.Didn't retrieve data . /1344106191.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106575', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106568', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106564', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106325', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106321', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106314', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106308', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106300', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106225', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106191', None, None, None, None, None, '2668799') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013009071350097656 save_final save missing photos in datou_result : time spend for datou_step_exec : 113.80240273475647 time spend to save output : 0.013725996017456055 total time spend for step 5 : 113.81612873077393 step6:crop_condition Fri Mar 14 11:26:33 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 139 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! About to upload 7 photos upload in portfolio : 4869462 init cache_photo without model_param we have 7 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741947997_3350784 we have uploaded 7 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.6239941120147705 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! About to upload 7 photos upload in portfolio : 4869462 init cache_photo without model_param we have 7 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741948002_3350784 we have uploaded 7 photos in the portfolio 4869462 time of upload the photos Elapsed time : 4.059988021850586 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741948008_3350784 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.5432794094085693 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741948011_3350784 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.5923783779144287 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741948014_3350784 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.0996553897857666 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741948018_3350784 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.559657335281372 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741948023_3350784 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.5385215282440186 we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1344106575, 1344106568, 1344106564, 1344106325, 1344106321, 1344106314, 1344106308, 1344106300, 1344106225, 1344106191] Looping around the photos to save general results len do output : 29 /1344117532Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117533Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117534Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117535Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117536Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117537Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117538Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117541Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117542Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117543Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117544Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117545Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117546Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117547Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117554Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117555Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117556Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117558Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117561Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117568Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117569Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117570Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117578Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117579Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117580Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117581Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117582Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117583Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1344117588Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106575', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106568', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106564', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106325', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106321', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106314', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106308', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106300', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106225', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106191', None, None, None, None, None, '2668799') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 97 time used for this insertion : 2.0273919105529785 save_final save missing photos in datou_result : time spend for datou_step_exec : 30.951160669326782 time spend to save output : 2.0285792350769043 total time spend for step 6 : 32.97973990440369 step7:ventilate_hashtags_in_portfolio Fri Mar 14 11:27:06 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 21421446 get user id for portfolio 21421446 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21421446 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('kraft','mal_croppe','Carton_brun','autre_refus','papier','Carton_gris','cartonnette','plastique','environnement','metal','flou','Teint_Dans_La_Masse')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21421446 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('kraft','mal_croppe','Carton_brun','autre_refus','papier','Carton_gris','cartonnette','plastique','environnement','metal','flou','Teint_Dans_La_Masse')) AND mptpi.`min_score`=0.5 To do To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21421446 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('kraft','mal_croppe','Carton_brun','autre_refus','papier','Carton_gris','cartonnette','plastique','environnement','metal','flou','Teint_Dans_La_Masse')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/21421649,21421650,21421651,21421652,21421653,21421654,21421655,21421656,21421657,21421658,21421659,21421660?tags=kraft,mal_croppe,Carton_brun,autre_refus,papier,Carton_gris,cartonnette,plastique,environnement,metal,flou,Teint_Dans_La_Masse Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1344106575, 1344106568, 1344106564, 1344106325, 1344106321, 1344106314, 1344106308, 1344106300, 1344106225, 1344106191] Looping around the photos to save general results len do output : 1 /21421446. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106575', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106568', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106564', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106325', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106321', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106314', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106308', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106300', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106225', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106191', None, None, None, None, None, '2668799') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.01360940933227539 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.2553062438964844 time spend to save output : 0.013877391815185547 total time spend for step 7 : 1.26918363571167 step8:final Fri Mar 14 11:27:07 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1344106575: ('0.06154173459865928',), 1344106568: ('0.06154173459865928',), 1344106564: ('0.06154173459865928',), 1344106325: ('0.06154173459865928',), 1344106321: ('0.06154173459865928',), 1344106314: ('0.06154173459865928',), 1344106308: ('0.06154173459865928',), 1344106300: ('0.06154173459865928',), 1344106225: ('0.06154173459865928',), 1344106191: ('0.06154173459865928',)} new output for save of step final : {1344106575: ('0.06154173459865928',), 1344106568: ('0.06154173459865928',), 1344106564: ('0.06154173459865928',), 1344106325: ('0.06154173459865928',), 1344106321: ('0.06154173459865928',), 1344106314: ('0.06154173459865928',), 1344106308: ('0.06154173459865928',), 1344106300: ('0.06154173459865928',), 1344106225: ('0.06154173459865928',), 1344106191: ('0.06154173459865928',)} [1344106575, 1344106568, 1344106564, 1344106325, 1344106321, 1344106314, 1344106308, 1344106300, 1344106225, 1344106191] Looping around the photos to save general results len do output : 10 /1344106575.Didn't retrieve data . /1344106568.Didn't retrieve data . /1344106564.Didn't retrieve data . /1344106325.Didn't retrieve data . /1344106321.Didn't retrieve data . /1344106314.Didn't retrieve data . /1344106308.Didn't retrieve data . /1344106300.Didn't retrieve data . /1344106225.Didn't retrieve data . /1344106191.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106575', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106568', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106564', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106325', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106321', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106314', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106308', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106300', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106225', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106191', None, None, None, None, None, '2668799') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.012749671936035156 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.20453095436096191 time spend to save output : 0.013473987579345703 total time spend for step 8 : 0.21800494194030762 step9:velours_tree Fri Mar 14 11:27:07 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.13566350936889648 time spend to save output : 8.034706115722656e-05 total time spend for step 9 : 0.1357438564300537 step10:send_mail_cod Fri Mar 14 11:27:07 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P21421446_14-03-2025_11_27_07.pdf 21421649 change filename to text .imagette214216491741948027 21421650 imagette214216501741948028 21421652 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette214216521741948028 21421655 change filename to text .imagette214216551741948028 21421656 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette214216561741948028 21421658 imagette214216581741948029 21421659 imagette214216591741948029 21421651 change filename to text .change filename to text .change filename to text .imagette214216511741948029 21421654 change filename to text .change filename to text .change filename to text .change filename to text .imagette214216541741948029 21421660 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette214216601741948030 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=21421446 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/21421649,21421650,21421651,21421652,21421653,21421654,21421655,21421656,21421657,21421658,21421659,21421660?tags=kraft,mal_croppe,Carton_brun,autre_refus,papier,Carton_gris,cartonnette,plastique,environnement,metal,flou,Teint_Dans_La_Masse your option no_mail is active, we will not send the real mail to your client args[1344106575] : ('0.06154173459865928',) no score found for photo 1344106575 We are sending mail with results at report@fotonower.com args[1344106568] : ('0.06154173459865928',) no score found for photo 1344106568 We are sending mail with results at report@fotonower.com args[1344106564] : ('0.06154173459865928',) no score found for photo 1344106564 We are sending mail with results at report@fotonower.com args[1344106325] : ('0.06154173459865928',) no score found for photo 1344106325 We are sending mail with results at report@fotonower.com args[1344106321] : ('0.06154173459865928',) no score found for photo 1344106321 We are sending mail with results at report@fotonower.com args[1344106314] : ('0.06154173459865928',) no score found for photo 1344106314 We are sending mail with results at report@fotonower.com args[1344106308] : ('0.06154173459865928',) no score found for photo 1344106308 We are sending mail with results at report@fotonower.com args[1344106300] : ('0.06154173459865928',) no score found for photo 1344106300 We are sending mail with results at report@fotonower.com args[1344106225] : ('0.06154173459865928',) no score found for photo 1344106225 We are sending mail with results at report@fotonower.com args[1344106191] : ('0.06154173459865928',) no score found for photo 1344106191 We are sending mail with results at report@fotonower.com refus_total : 0.06154173459865928 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=21421446 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1344105212,1344106325,1344106308,1344106225,1344106191,1344106186,1344106136,1344106134,1344105226,1344106314,1344106141,1344106179,1344106131,1344105353,1344106300,1344106321,1344105153,1344106564,1344106568,1344106575) Found this number of photos: 20 begin to download photo : 1344105212 begin to download photo : 1344106186 begin to download photo : 1344106141 begin to download photo : 1344106321 download finish for photo 1344105212 begin to download photo : 1344106325 download finish for photo 1344106321 begin to download photo : 1344105153 download finish for photo 1344106141 begin to download photo : 1344106179 download finish for photo 1344106186 begin to download photo : 1344106136 download finish for photo 1344106179 begin to download photo : 1344106131 download finish for photo 1344106136 begin to download photo : 1344106134 download finish for photo 1344105153 begin to download photo : 1344106564 download finish for photo 1344106325 begin to download photo : 1344106308 download finish for photo 1344106134 begin to download photo : 1344105226 download finish for photo 1344106564 begin to download photo : 1344106568 download finish for photo 1344106308 begin to download photo : 1344106225 download finish for photo 1344106131 begin to download photo : 1344105353 download finish for photo 1344105226 begin to download photo : 1344106314 download finish for photo 1344106225 begin to download photo : 1344106191 download finish for photo 1344105353 begin to download photo : 1344106300 download finish for photo 1344106568 begin to download photo : 1344106575 download finish for photo 1344106191 download finish for photo 1344106314 download finish for photo 1344106300 download finish for photo 1344106575 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21421446_14-03-2025_11_27_07.pdf results_Auto_P21421446_14-03-2025_11_27_07.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21421446_14-03-2025_11_27_07.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','21421446','results_Auto_P21421446_14-03-2025_11_27_07.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21421446_14-03-2025_11_27_07.pdf','pdf','','0.7','0.06154173459865928') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1344106575, 1344106568, 1344106564, 1344106325, 1344106321, 1344106314, 1344106308, 1344106300, 1344106225, 1344106191] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106575', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106568', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106564', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106325', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106321', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106314', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106308', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106300', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106225', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106191', None, None, None, None, None, '2668799') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.01431131362915039 save_final save missing photos in datou_result : time spend for datou_step_exec : 6.384754419326782 time spend to save output : 0.014851093292236328 total time spend for step 10 : 6.3996055126190186 step11:split_time_score Fri Mar 14 11:27:14 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('10', 20),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 14032025 21421446 Nombre de photos uploadées : 20 / 23040 (0%) 14032025 21421446 Nombre de photos taguées (types de déchets): 0 / 20 (0%) 14032025 21421446 Nombre de photos taguées (volume) : 0 / 20 (0%) elapsed_time : load_data_split_time_score 4.5299530029296875e-06 elapsed_time : order_list_meta_photo_and_scores 1.7404556274414062e-05 ???????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0015692710876464844 elapsed_time : insert_dashboard_record_day_entry 0.02231121063232422 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.05674420462057292 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21416179_14-03-2025_09_50_12.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21416179 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21416179 AND mptpi.`type`=4207 To do Qualite : 0.17794234731237804 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21416180_14-03-2025_09_07_29.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21416180 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21416180 AND mptpi.`type`=4207 To do Qualite : 0.20334743529613492 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21416181_14-03-2025_09_29_00.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21416181 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21416181 AND mptpi.`type`=3594 To do Qualite : 0.1607797574417757 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21416185_14-03-2025_10_02_36.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21416185 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21416185 AND mptpi.`type`=4207 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21416200 order by id desc limit 1 Qualite : 0.11110619430725803 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21416201_14-03-2025_09_35_56.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21416201 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21416201 AND mptpi.`type`=4207 To do Qualite : 0.15792279568209802 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21416202_14-03-2025_09_23_19.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21416202 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21416202 AND mptpi.`type`=4207 To do Qualite : 0.03991597164895486 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21418608_14-03-2025_10_22_58.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21418608 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21418608 AND mptpi.`type`=4207 To do Qualite : 0.04468489596863188 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21418609_14-03-2025_11_03_07.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21418609 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21418609 AND mptpi.`type`=4207 To do Qualite : 0.4241951561233941 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21418610_14-03-2025_10_57_34.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21418610 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11548 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11556 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11557 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11552 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11552 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11555 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11555 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11549 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11580 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11580 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11551 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11550 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11550 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11559 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11551 doesn't seem to be define in the database( WARNING : type of input 3 of step 11550 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11548 doesn't seem to be define in the database( WARNING : type of input 2 of step 11552 doesn't seem to be define in the database( WARNING : output 1 of step 11548 have datatype=2 whereas input 1 of step 11555 have datatype=7 WARNING : type of output 2 of step 11555 doesn't seem to be define in the database( WARNING : type of input 1 of step 11549 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11551 have datatype=10 whereas input 3 of step 11558 have datatype=6 WARNING : type of input 2 of step 11580 doesn't seem to be define in the database( WARNING : output 1 of step 11549 have datatype=7 whereas input 2 of step 11580 have datatype=None WARNING : type of output 3 of step 11580 doesn't seem to be define in the database( WARNING : type of input 1 of step 11551 doesn't seem to be define in the database( WARNING : output 0 of step 11551 have datatype=10 whereas input 0 of step 11586 have datatype=18 WARNING : type of input 5 of step 11558 doesn't seem to be define in the database( WARNING : output 0 of step 11586 have datatype=11 whereas input 5 of step 11558 have datatype=None WARNING : type of output 1 of step 11556 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : type of output 1 of step 11557 doesn't seem to be define in the database( WARNING : type of input 3 of step 11552 doesn't seem to be define in the database( WARNING : output 0 of step 11555 have datatype=1 whereas input 0 of step 11549 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21418610 AND mptpi.`type`=4200 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21421445 order by id desc limit 1 Qualite : 0.06154173459865928 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21421446_14-03-2025_11_27_07.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21421446 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21421446 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21421447 order by id desc limit 1 NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'14032025': {'nb_upload': 20, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1344106575, 1344106568, 1344106564, 1344106325, 1344106321, 1344106314, 1344106308, 1344106300, 1344106225, 1344106191] Looping around the photos to save general results len do output : 1 /21421446Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106575', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106568', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106564', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106325', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106321', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106314', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106308', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106300', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106225', None, None, None, None, None, '2668799') ('3459', None, None, None, None, None, None, None, '2668799') ('3459', '21421446', '1344106191', None, None, None, None, None, '2668799') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.013017416000366211 save_final save missing photos in datou_result : time spend for datou_step_exec : 3.168736696243286 time spend to save output : 0.013242483139038086 total time spend for step 11 : 3.181979179382324 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 146.04user 122.43system 4:54.36elapsed 91%CPU (0avgtext+0avgdata 4229216maxresident)k 540464inputs+114136outputs (12627major+12565420minor)pagefaults 0swaps