python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 7911 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 28>10 please execute split_portfolio.py -i 20461712 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2577764'] with mtr_portfolio_ids : ['20461712'] and first list_photo_ids : [] new path : /proc/7911/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 1.9522504806518555 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Tue Feb 11 21:12:31 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5043 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-11 21:12:35.110422: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-11 21:12:35.159107: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-11 21:12:35.160924: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f069c000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-11 21:12:35.161002: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-11 21:12:35.167714: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-11 21:12:35.293163: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x41bfb490 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-11 21:12:35.293225: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-11 21:12:35.294510: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-11 21:12:35.295003: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-11 21:12:35.298151: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-11 21:12:35.302379: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-11 21:12:35.302788: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-11 21:12:35.310810: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-11 21:12:35.313218: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-11 21:12:35.330917: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-11 21:12:35.332430: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-11 21:12:35.332542: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-11 21:12:35.333319: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-11 21:12:35.333338: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-11 21:12:35.333350: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-11 21:12:35.334617: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4591 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-11 21:12:35.716527: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-11 21:12:35.716683: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-11 21:12:35.716706: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-11 21:12:35.716725: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-11 21:12:35.716742: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-11 21:12:35.716760: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-11 21:12:35.716777: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-11 21:12:35.716795: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-11 21:12:35.718022: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-11 21:12:35.719597: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-11 21:12:35.719655: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-11 21:12:35.719678: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-11 21:12:35.719700: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-11 21:12:35.719719: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-11 21:12:35.719737: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-11 21:12:35.719754: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-11 21:12:35.719770: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-11 21:12:35.721169: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-11 21:12:35.721230: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-11 21:12:35.721249: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-11 21:12:35.721265: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-11 21:12:35.722880: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4591 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-11 21:12:42.337587: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 51380224 exceeds 10% of free system memory. 2025-02-11 21:12:48.730760: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-11 21:12:48.985703: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-11 21:12:50.992762: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.98G (4279107584 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 19 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 11 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 25 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 21 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 15 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 24 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 26 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 24 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 6 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 17 Detection mask done ! Trying to reset tf kernel 8467 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 176 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5043 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.007951736450195312 nb_pixel_total : 173571 time to create 1 rle with new method : 0.014534711837768555 length of segment : 378 time for calcul the mask position with numpy : 0.0026209354400634766 nb_pixel_total : 106789 time to create 1 rle with old method : 0.12557601928710938 length of segment : 627 time for calcul the mask position with numpy : 0.00013208389282226562 nb_pixel_total : 1800 time to create 1 rle with old method : 0.002302408218383789 length of segment : 43 time for calcul the mask position with numpy : 0.0006442070007324219 nb_pixel_total : 26231 time to create 1 rle with old method : 0.029512405395507812 length of segment : 159 time for calcul the mask position with numpy : 0.00022411346435546875 nb_pixel_total : 8153 time to create 1 rle with old method : 0.010212182998657227 length of segment : 95 time for calcul the mask position with numpy : 0.0005483627319335938 nb_pixel_total : 30509 time to create 1 rle with old method : 0.034754037857055664 length of segment : 180 time for calcul the mask position with numpy : 0.000385284423828125 nb_pixel_total : 7842 time to create 1 rle with old method : 0.009031295776367188 length of segment : 139 time for calcul the mask position with numpy : 0.00012302398681640625 nb_pixel_total : 3197 time to create 1 rle with old method : 0.00405430793762207 length of segment : 74 time for calcul the mask position with numpy : 0.0016446113586425781 nb_pixel_total : 75700 time to create 1 rle with old method : 0.08705973625183105 length of segment : 354 time for calcul the mask position with numpy : 0.0009582042694091797 nb_pixel_total : 33097 time to create 1 rle with old method : 0.051267147064208984 length of segment : 225 time for calcul the mask position with numpy : 0.0017025470733642578 nb_pixel_total : 87319 time to create 1 rle with old method : 0.09865236282348633 length of segment : 355 time for calcul the mask position with numpy : 0.00032138824462890625 nb_pixel_total : 7222 time to create 1 rle with old method : 0.008534908294677734 length of segment : 89 time for calcul the mask position with numpy : 0.0006012916564941406 nb_pixel_total : 17874 time to create 1 rle with old method : 0.02002096176147461 length of segment : 265 time for calcul the mask position with numpy : 0.0008766651153564453 nb_pixel_total : 34187 time to create 1 rle with old method : 0.03953266143798828 length of segment : 273 time for calcul the mask position with numpy : 0.0018162727355957031 nb_pixel_total : 63028 time to create 1 rle with old method : 0.06897091865539551 length of segment : 348 time for calcul the mask position with numpy : 0.0012922286987304688 nb_pixel_total : 45536 time to create 1 rle with old method : 0.05074620246887207 length of segment : 407 time for calcul the mask position with numpy : 0.005445241928100586 nb_pixel_total : 183002 time to create 1 rle with new method : 0.012465476989746094 length of segment : 523 time for calcul the mask position with numpy : 0.003245830535888672 nb_pixel_total : 142005 time to create 1 rle with old method : 0.1716153621673584 length of segment : 577 time for calcul the mask position with numpy : 0.003798246383666992 nb_pixel_total : 194394 time to create 1 rle with new method : 0.015118837356567383 length of segment : 627 time for calcul the mask position with numpy : 0.00044989585876464844 nb_pixel_total : 13145 time to create 1 rle with old method : 0.015458345413208008 length of segment : 215 time for calcul the mask position with numpy : 0.0031309127807617188 nb_pixel_total : 139807 time to create 1 rle with old method : 0.15993380546569824 length of segment : 737 time for calcul the mask position with numpy : 0.0001537799835205078 nb_pixel_total : 3051 time to create 1 rle with old method : 0.003655672073364258 length of segment : 47 time for calcul the mask position with numpy : 0.0020089149475097656 nb_pixel_total : 47988 time to create 1 rle with old method : 0.053849220275878906 length of segment : 514 time for calcul the mask position with numpy : 0.0007336139678955078 nb_pixel_total : 12145 time to create 1 rle with old method : 0.0141143798828125 length of segment : 196 time for calcul the mask position with numpy : 0.0001552104949951172 nb_pixel_total : 4038 time to create 1 rle with old method : 0.005055904388427734 length of segment : 64 time for calcul the mask position with numpy : 0.00015401840209960938 nb_pixel_total : 2657 time to create 1 rle with old method : 0.0031232833862304688 length of segment : 82 time for calcul the mask position with numpy : 0.00043082237243652344 nb_pixel_total : 17546 time to create 1 rle with old method : 0.020317792892456055 length of segment : 272 time for calcul the mask position with numpy : 0.00014781951904296875 nb_pixel_total : 2092 time to create 1 rle with old method : 0.002638101577758789 length of segment : 54 time for calcul the mask position with numpy : 0.0017826557159423828 nb_pixel_total : 68780 time to create 1 rle with old method : 0.07903432846069336 length of segment : 476 time for calcul the mask position with numpy : 0.018307209014892578 nb_pixel_total : 342244 time to create 1 rle with new method : 0.029496431350708008 length of segment : 838 time for calcul the mask position with numpy : 0.002234220504760742 nb_pixel_total : 38838 time to create 1 rle with old method : 0.04427599906921387 length of segment : 330 time for calcul the mask position with numpy : 0.003658294677734375 nb_pixel_total : 102086 time to create 1 rle with old method : 0.1186368465423584 length of segment : 449 time for calcul the mask position with numpy : 0.0005018711090087891 nb_pixel_total : 7403 time to create 1 rle with old method : 0.008562326431274414 length of segment : 124 time for calcul the mask position with numpy : 0.00033402442932128906 nb_pixel_total : 5828 time to create 1 rle with old method : 0.006911277770996094 length of segment : 109 time for calcul the mask position with numpy : 0.0032842159271240234 nb_pixel_total : 47278 time to create 1 rle with old method : 0.11281013488769531 length of segment : 255 time for calcul the mask position with numpy : 0.008226156234741211 nb_pixel_total : 236506 time to create 1 rle with new method : 0.47108983993530273 length of segment : 545 time for calcul the mask position with numpy : 0.0005631446838378906 nb_pixel_total : 11097 time to create 1 rle with old method : 0.012658834457397461 length of segment : 100 time for calcul the mask position with numpy : 0.0009715557098388672 nb_pixel_total : 11895 time to create 1 rle with old method : 0.015743732452392578 length of segment : 253 time for calcul the mask position with numpy : 0.0022683143615722656 nb_pixel_total : 44069 time to create 1 rle with old method : 0.05366182327270508 length of segment : 347 time for calcul the mask position with numpy : 0.001623392105102539 nb_pixel_total : 35164 time to create 1 rle with old method : 0.046330928802490234 length of segment : 289 time for calcul the mask position with numpy : 0.005209922790527344 nb_pixel_total : 94997 time to create 1 rle with old method : 0.1175999641418457 length of segment : 475 time for calcul the mask position with numpy : 0.0012774467468261719 nb_pixel_total : 38303 time to create 1 rle with old method : 0.04383087158203125 length of segment : 248 time for calcul the mask position with numpy : 0.0012888908386230469 nb_pixel_total : 7947 time to create 1 rle with old method : 0.01069021224975586 length of segment : 243 time for calcul the mask position with numpy : 0.0006606578826904297 nb_pixel_total : 22541 time to create 1 rle with old method : 0.027795791625976562 length of segment : 176 time for calcul the mask position with numpy : 0.0002903938293457031 nb_pixel_total : 3781 time to create 1 rle with old method : 0.0048084259033203125 length of segment : 137 time for calcul the mask position with numpy : 0.0019309520721435547 nb_pixel_total : 55026 time to create 1 rle with old method : 0.06304430961608887 length of segment : 245 time for calcul the mask position with numpy : 0.010618448257446289 nb_pixel_total : 356034 time to create 1 rle with new method : 0.13275599479675293 length of segment : 491 time for calcul the mask position with numpy : 0.0004317760467529297 nb_pixel_total : 4105 time to create 1 rle with old method : 0.004931449890136719 length of segment : 132 time for calcul the mask position with numpy : 0.0017626285552978516 nb_pixel_total : 40462 time to create 1 rle with old method : 0.044527292251586914 length of segment : 363 time for calcul the mask position with numpy : 0.002790212631225586 nb_pixel_total : 102695 time to create 1 rle with old method : 0.15089178085327148 length of segment : 487 time for calcul the mask position with numpy : 0.0010113716125488281 nb_pixel_total : 53404 time to create 1 rle with old method : 0.061437368392944336 length of segment : 244 time for calcul the mask position with numpy : 0.0001785755157470703 nb_pixel_total : 1893 time to create 1 rle with old method : 0.002457141876220703 length of segment : 48 time for calcul the mask position with numpy : 0.0002319812774658203 nb_pixel_total : 1813 time to create 1 rle with old method : 0.0021889209747314453 length of segment : 85 time for calcul the mask position with numpy : 0.0006363391876220703 nb_pixel_total : 12930 time to create 1 rle with old method : 0.015563249588012695 length of segment : 123 time for calcul the mask position with numpy : 0.0004794597625732422 nb_pixel_total : 10535 time to create 1 rle with old method : 0.01318812370300293 length of segment : 126 time for calcul the mask position with numpy : 0.006812334060668945 nb_pixel_total : 159070 time to create 1 rle with new method : 0.013334512710571289 length of segment : 566 time for calcul the mask position with numpy : 0.0019178390502929688 nb_pixel_total : 34328 time to create 1 rle with old method : 0.042830467224121094 length of segment : 422 time for calcul the mask position with numpy : 0.009663581848144531 nb_pixel_total : 178831 time to create 1 rle with new method : 0.04767870903015137 length of segment : 1122 time for calcul the mask position with numpy : 0.0012004375457763672 nb_pixel_total : 38292 time to create 1 rle with old method : 0.04565095901489258 length of segment : 315 time for calcul the mask position with numpy : 0.0001285076141357422 nb_pixel_total : 2723 time to create 1 rle with old method : 0.003347158432006836 length of segment : 61 time for calcul the mask position with numpy : 0.028586864471435547 nb_pixel_total : 588717 time to create 1 rle with new method : 0.30086612701416016 length of segment : 1125 time for calcul the mask position with numpy : 0.0005438327789306641 nb_pixel_total : 11325 time to create 1 rle with old method : 0.013669013977050781 length of segment : 148 time for calcul the mask position with numpy : 0.0004055500030517578 nb_pixel_total : 17747 time to create 1 rle with old method : 0.020807266235351562 length of segment : 287 time for calcul the mask position with numpy : 0.00019121170043945312 nb_pixel_total : 4854 time to create 1 rle with old method : 0.006796121597290039 length of segment : 84 time for calcul the mask position with numpy : 0.0010094642639160156 nb_pixel_total : 34016 time to create 1 rle with old method : 0.03748178482055664 length of segment : 229 time for calcul the mask position with numpy : 0.0003879070281982422 nb_pixel_total : 12004 time to create 1 rle with old method : 0.014232397079467773 length of segment : 108 time for calcul the mask position with numpy : 0.0003268718719482422 nb_pixel_total : 6741 time to create 1 rle with old method : 0.00837850570678711 length of segment : 123 time for calcul the mask position with numpy : 0.0021123886108398438 nb_pixel_total : 53969 time to create 1 rle with old method : 0.06086993217468262 length of segment : 757 time for calcul the mask position with numpy : 0.0003311634063720703 nb_pixel_total : 5485 time to create 1 rle with old method : 0.006908416748046875 length of segment : 90 time for calcul the mask position with numpy : 0.3004028797149658 nb_pixel_total : 660717 time to create 1 rle with new method : 0.15508413314819336 length of segment : 1104 time for calcul the mask position with numpy : 0.001245737075805664 nb_pixel_total : 41392 time to create 1 rle with old method : 0.04622364044189453 length of segment : 240 time for calcul the mask position with numpy : 0.0006897449493408203 nb_pixel_total : 9025 time to create 1 rle with old method : 0.016460895538330078 length of segment : 121 time for calcul the mask position with numpy : 0.0003495216369628906 nb_pixel_total : 6230 time to create 1 rle with old method : 0.007943153381347656 length of segment : 112 time for calcul the mask position with numpy : 0.0007464885711669922 nb_pixel_total : 24595 time to create 1 rle with old method : 0.029011249542236328 length of segment : 205 time for calcul the mask position with numpy : 0.0018572807312011719 nb_pixel_total : 40610 time to create 1 rle with old method : 0.05191302299499512 length of segment : 306 time for calcul the mask position with numpy : 0.007189512252807617 nb_pixel_total : 154776 time to create 1 rle with new method : 0.009832143783569336 length of segment : 570 time for calcul the mask position with numpy : 0.00026416778564453125 nb_pixel_total : 3989 time to create 1 rle with old method : 0.005030393600463867 length of segment : 107 time for calcul the mask position with numpy : 0.0018551349639892578 nb_pixel_total : 38820 time to create 1 rle with old method : 0.0417943000793457 length of segment : 543 time for calcul the mask position with numpy : 0.0023839473724365234 nb_pixel_total : 123862 time to create 1 rle with old method : 0.13927125930786133 length of segment : 404 time for calcul the mask position with numpy : 0.0018768310546875 nb_pixel_total : 112449 time to create 1 rle with old method : 0.13393950462341309 length of segment : 704 time for calcul the mask position with numpy : 0.00037598609924316406 nb_pixel_total : 3876 time to create 1 rle with old method : 0.0047724246978759766 length of segment : 99 time for calcul the mask position with numpy : 0.002793550491333008 nb_pixel_total : 94263 time to create 1 rle with old method : 0.10482001304626465 length of segment : 365 time for calcul the mask position with numpy : 0.00018668174743652344 nb_pixel_total : 2018 time to create 1 rle with old method : 0.0027146339416503906 length of segment : 76 time for calcul the mask position with numpy : 0.0009837150573730469 nb_pixel_total : 19002 time to create 1 rle with old method : 0.02239704132080078 length of segment : 183 time for calcul the mask position with numpy : 0.0063190460205078125 nb_pixel_total : 160853 time to create 1 rle with new method : 0.009743928909301758 length of segment : 822 time for calcul the mask position with numpy : 0.00933384895324707 nb_pixel_total : 309156 time to create 1 rle with new method : 0.01347494125366211 length of segment : 929 time for calcul the mask position with numpy : 0.00039887428283691406 nb_pixel_total : 4827 time to create 1 rle with old method : 0.008092164993286133 length of segment : 96 time for calcul the mask position with numpy : 0.007404804229736328 nb_pixel_total : 189295 time to create 1 rle with new method : 0.014391183853149414 length of segment : 723 time for calcul the mask position with numpy : 0.0005664825439453125 nb_pixel_total : 11733 time to create 1 rle with old method : 0.014122486114501953 length of segment : 107 time for calcul the mask position with numpy : 0.0024514198303222656 nb_pixel_total : 87008 time to create 1 rle with old method : 0.09483480453491211 length of segment : 468 time for calcul the mask position with numpy : 0.0010292530059814453 nb_pixel_total : 30865 time to create 1 rle with old method : 0.03746771812438965 length of segment : 182 time for calcul the mask position with numpy : 0.0001723766326904297 nb_pixel_total : 2288 time to create 1 rle with old method : 0.002835512161254883 length of segment : 90 time for calcul the mask position with numpy : 0.001619577407836914 nb_pixel_total : 46078 time to create 1 rle with old method : 0.06184124946594238 length of segment : 260 time for calcul the mask position with numpy : 0.0002048015594482422 nb_pixel_total : 839 time to create 1 rle with old method : 0.0011060237884521484 length of segment : 64 time for calcul the mask position with numpy : 0.0016841888427734375 nb_pixel_total : 17170 time to create 1 rle with old method : 0.02057790756225586 length of segment : 295 time for calcul the mask position with numpy : 0.0009245872497558594 nb_pixel_total : 42411 time to create 1 rle with old method : 0.0513606071472168 length of segment : 304 time for calcul the mask position with numpy : 0.0043792724609375 nb_pixel_total : 155535 time to create 1 rle with new method : 0.007424116134643555 length of segment : 558 time for calcul the mask position with numpy : 0.0017502307891845703 nb_pixel_total : 94190 time to create 1 rle with old method : 0.10022878646850586 length of segment : 379 time for calcul the mask position with numpy : 0.006096601486206055 nb_pixel_total : 203758 time to create 1 rle with new method : 0.009953975677490234 length of segment : 364 time for calcul the mask position with numpy : 0.003645658493041992 nb_pixel_total : 119741 time to create 1 rle with old method : 0.12813425064086914 length of segment : 362 time for calcul the mask position with numpy : 0.005497455596923828 nb_pixel_total : 268003 time to create 1 rle with new method : 0.014508724212646484 length of segment : 561 time for calcul the mask position with numpy : 0.0013799667358398438 nb_pixel_total : 25280 time to create 1 rle with old method : 0.028529882431030273 length of segment : 216 time for calcul the mask position with numpy : 0.0005807876586914062 nb_pixel_total : 10400 time to create 1 rle with old method : 0.014642953872680664 length of segment : 145 time for calcul the mask position with numpy : 0.0009219646453857422 nb_pixel_total : 20292 time to create 1 rle with old method : 0.025137662887573242 length of segment : 176 time for calcul the mask position with numpy : 0.00023365020751953125 nb_pixel_total : 2371 time to create 1 rle with old method : 0.003010272979736328 length of segment : 74 time for calcul the mask position with numpy : 0.0060062408447265625 nb_pixel_total : 125414 time to create 1 rle with old method : 0.13463187217712402 length of segment : 440 time for calcul the mask position with numpy : 0.0015735626220703125 nb_pixel_total : 28124 time to create 1 rle with old method : 0.030620098114013672 length of segment : 328 time for calcul the mask position with numpy : 0.0028007030487060547 nb_pixel_total : 92148 time to create 1 rle with old method : 0.10096526145935059 length of segment : 251 time for calcul the mask position with numpy : 0.0044596195220947266 nb_pixel_total : 104003 time to create 1 rle with old method : 0.11219096183776855 length of segment : 468 time for calcul the mask position with numpy : 0.0051267147064208984 nb_pixel_total : 166619 time to create 1 rle with new method : 0.010168790817260742 length of segment : 370 time for calcul the mask position with numpy : 0.0003070831298828125 nb_pixel_total : 9423 time to create 1 rle with old method : 0.010645866394042969 length of segment : 107 time for calcul the mask position with numpy : 0.0037348270416259766 nb_pixel_total : 85063 time to create 1 rle with old method : 0.08924007415771484 length of segment : 726 time for calcul the mask position with numpy : 0.0002589225769042969 nb_pixel_total : 3835 time to create 1 rle with old method : 0.0048503875732421875 length of segment : 49 time for calcul the mask position with numpy : 0.0002582073211669922 nb_pixel_total : 4236 time to create 1 rle with old method : 0.00522160530090332 length of segment : 78 time for calcul the mask position with numpy : 0.0006115436553955078 nb_pixel_total : 12212 time to create 1 rle with old method : 0.01369786262512207 length of segment : 132 time for calcul the mask position with numpy : 0.002574443817138672 nb_pixel_total : 65740 time to create 1 rle with old method : 0.08067488670349121 length of segment : 387 time for calcul the mask position with numpy : 0.03052520751953125 nb_pixel_total : 716871 time to create 1 rle with new method : 0.27988505363464355 length of segment : 1222 time for calcul the mask position with numpy : 0.0006821155548095703 nb_pixel_total : 11617 time to create 1 rle with old method : 0.019687175750732422 length of segment : 147 time for calcul the mask position with numpy : 0.0013453960418701172 nb_pixel_total : 25593 time to create 1 rle with old method : 0.02908635139465332 length of segment : 236 time for calcul the mask position with numpy : 0.026528120040893555 nb_pixel_total : 854265 time to create 1 rle with new method : 0.357379674911499 length of segment : 1358 time for calcul the mask position with numpy : 0.00031185150146484375 nb_pixel_total : 2878 time to create 1 rle with old method : 0.003820657730102539 length of segment : 60 time for calcul the mask position with numpy : 0.00046515464782714844 nb_pixel_total : 8025 time to create 1 rle with old method : 0.009585380554199219 length of segment : 152 time for calcul the mask position with numpy : 0.0027680397033691406 nb_pixel_total : 91258 time to create 1 rle with old method : 0.10115885734558105 length of segment : 256 time for calcul the mask position with numpy : 0.005710601806640625 nb_pixel_total : 208268 time to create 1 rle with new method : 0.014794111251831055 length of segment : 630 time for calcul the mask position with numpy : 0.0021042823791503906 nb_pixel_total : 52898 time to create 1 rle with old method : 0.0599513053894043 length of segment : 226 time for calcul the mask position with numpy : 0.005668163299560547 nb_pixel_total : 47633 time to create 1 rle with old method : 0.07319855690002441 length of segment : 621 time for calcul the mask position with numpy : 0.0021796226501464844 nb_pixel_total : 63256 time to create 1 rle with old method : 0.07224345207214355 length of segment : 407 time for calcul the mask position with numpy : 0.000247955322265625 nb_pixel_total : 3526 time to create 1 rle with old method : 0.005382537841796875 length of segment : 55 time for calcul the mask position with numpy : 0.004373788833618164 nb_pixel_total : 154450 time to create 1 rle with new method : 0.006129264831542969 length of segment : 415 time for calcul the mask position with numpy : 0.0010197162628173828 nb_pixel_total : 22003 time to create 1 rle with old method : 0.027054309844970703 length of segment : 316 time for calcul the mask position with numpy : 0.0007236003875732422 nb_pixel_total : 8662 time to create 1 rle with old method : 0.010942697525024414 length of segment : 166 time for calcul the mask position with numpy : 0.0005383491516113281 nb_pixel_total : 8075 time to create 1 rle with old method : 0.009819984436035156 length of segment : 136 time for calcul the mask position with numpy : 0.004747629165649414 nb_pixel_total : 152537 time to create 1 rle with new method : 0.007666110992431641 length of segment : 412 time for calcul the mask position with numpy : 0.0013301372528076172 nb_pixel_total : 43133 time to create 1 rle with old method : 0.046694278717041016 length of segment : 330 time for calcul the mask position with numpy : 0.0024831295013427734 nb_pixel_total : 104160 time to create 1 rle with old method : 0.11273360252380371 length of segment : 449 time for calcul the mask position with numpy : 0.0013852119445800781 nb_pixel_total : 29454 time to create 1 rle with old method : 0.03659701347351074 length of segment : 179 time for calcul the mask position with numpy : 0.0008945465087890625 nb_pixel_total : 48590 time to create 1 rle with old method : 0.05351853370666504 length of segment : 250 time for calcul the mask position with numpy : 0.0026123523712158203 nb_pixel_total : 124064 time to create 1 rle with old method : 0.13297700881958008 length of segment : 530 time for calcul the mask position with numpy : 0.014230489730834961 nb_pixel_total : 369945 time to create 1 rle with new method : 0.04047679901123047 length of segment : 939 time for calcul the mask position with numpy : 0.0007240772247314453 nb_pixel_total : 10787 time to create 1 rle with old method : 0.01281595230102539 length of segment : 147 time for calcul the mask position with numpy : 0.0002071857452392578 nb_pixel_total : 3715 time to create 1 rle with old method : 0.0048122406005859375 length of segment : 108 time for calcul the mask position with numpy : 0.001130819320678711 nb_pixel_total : 39947 time to create 1 rle with old method : 0.044764041900634766 length of segment : 329 time for calcul the mask position with numpy : 0.003557920455932617 nb_pixel_total : 100583 time to create 1 rle with old method : 0.1141211986541748 length of segment : 405 time for calcul the mask position with numpy : 0.0004608631134033203 nb_pixel_total : 7399 time to create 1 rle with old method : 0.009100675582885742 length of segment : 158 time for calcul the mask position with numpy : 0.0025031566619873047 nb_pixel_total : 99649 time to create 1 rle with old method : 0.1141824722290039 length of segment : 341 time for calcul the mask position with numpy : 0.0004923343658447266 nb_pixel_total : 10403 time to create 1 rle with old method : 0.012482643127441406 length of segment : 182 time for calcul the mask position with numpy : 0.5980865955352783 nb_pixel_total : 4372568 time to create 1 rle with new method : 0.6211962699890137 length of segment : 4527 time for calcul the mask position with numpy : 0.00027489662170410156 nb_pixel_total : 5064 time to create 1 rle with old method : 0.008499860763549805 length of segment : 73 time for calcul the mask position with numpy : 0.0002105236053466797 nb_pixel_total : 7768 time to create 1 rle with old method : 0.011106491088867188 length of segment : 81 time for calcul the mask position with numpy : 0.0002281665802001953 nb_pixel_total : 9722 time to create 1 rle with old method : 0.01156926155090332 length of segment : 130 time for calcul the mask position with numpy : 0.00439906120300293 nb_pixel_total : 319952 time to create 1 rle with new method : 0.008665800094604492 length of segment : 784 time for calcul the mask position with numpy : 0.0027272701263427734 nb_pixel_total : 98984 time to create 1 rle with old method : 0.1117243766784668 length of segment : 297 time for calcul the mask position with numpy : 0.00045800209045410156 nb_pixel_total : 8680 time to create 1 rle with old method : 0.010317087173461914 length of segment : 139 time for calcul the mask position with numpy : 0.000164031982421875 nb_pixel_total : 5156 time to create 1 rle with old method : 0.00633692741394043 length of segment : 83 time for calcul the mask position with numpy : 0.2649211883544922 nb_pixel_total : 1962374 time to create 1 rle with new method : 0.17562341690063477 length of segment : 3049 time for calcul the mask position with numpy : 0.001028299331665039 nb_pixel_total : 61659 time to create 1 rle with old method : 0.06777644157409668 length of segment : 328 time for calcul the mask position with numpy : 0.002463817596435547 nb_pixel_total : 56410 time to create 1 rle with old method : 0.07041001319885254 length of segment : 534 time for calcul the mask position with numpy : 0.01114964485168457 nb_pixel_total : 433123 time to create 1 rle with new method : 0.019907474517822266 length of segment : 1099 time for calcul the mask position with numpy : 0.0029592514038085938 nb_pixel_total : 104377 time to create 1 rle with old method : 0.12310051918029785 length of segment : 552 time for calcul the mask position with numpy : 0.00040411949157714844 nb_pixel_total : 4171 time to create 1 rle with old method : 0.009042501449584961 length of segment : 105 time for calcul the mask position with numpy : 0.0015413761138916016 nb_pixel_total : 71429 time to create 1 rle with old method : 0.08420228958129883 length of segment : 319 time for calcul the mask position with numpy : 0.0077838897705078125 nb_pixel_total : 41005 time to create 1 rle with old method : 0.05001235008239746 length of segment : 337 time for calcul the mask position with numpy : 0.0017979145050048828 nb_pixel_total : 63376 time to create 1 rle with old method : 0.07729458808898926 length of segment : 373 time for calcul the mask position with numpy : 0.01125788688659668 nb_pixel_total : 387011 time to create 1 rle with new method : 0.018523216247558594 length of segment : 1000 time for calcul the mask position with numpy : 0.0009069442749023438 nb_pixel_total : 18859 time to create 1 rle with old method : 0.021070003509521484 length of segment : 256 time for calcul the mask position with numpy : 0.0045320987701416016 nb_pixel_total : 171246 time to create 1 rle with new method : 0.006980180740356445 length of segment : 386 time spent for convertir_results : 22.86216139793396 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 166 chid ids of type : 3663 Number RLEs to save : 62358 save missing photos in datou_result : time spend for datou_step_exec : 84.7200255393982 time spend to save output : 3.5324292182922363 total time spend for step 1 : 88.25245475769043 step2:crop_condition Tue Feb 11 21:13:59 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 166 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 13 About to insert : list_path_to_insert length 13 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 16 About to insert : list_path_to_insert length 16 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 45 About to insert : list_path_to_insert length 45 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 11 About to insert : list_path_to_insert length 11 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1336885647, 1336885641, 1336885604, 1336885601, 1336885562, 1336885512, 1336885502, 1336885496, 1336885135, 1336884924] Looping around the photos to save general results len do output : 97 /-3670624560Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624571Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624586Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624600Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624591Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624590Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624622Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624635Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624691Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624689Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624708Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624718Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624711Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624568Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624563Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624578Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624580Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624634Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624652Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624649Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624682Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624684Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624671Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624704Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624696Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624701Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624723Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624724Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624725Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624610Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624613Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624615Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624641Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624665Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624722Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624573Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624574Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624570Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624575Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624577Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624572Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624576Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624569Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624589Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624582Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624588Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624602Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624603Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624599Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624605Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624606Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624598Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624594Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624608Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624618Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624617Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624627Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624639Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624637Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624630Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624644Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624645Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624659Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624658Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624656Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624661Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624675Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624681Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624676Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624685Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624678Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624669Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624698Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624702Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624688Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624693Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624697Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624721Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624720Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624715Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624561Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624579Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624687Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624695Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624699Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624716Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624609Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624624Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624638Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624657Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624647Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624660Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624655Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624667Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624692Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624710Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3670624709Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885647', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885641', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885604', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885601', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885562', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885512', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885502', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885496', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885135', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336884924', None, None, None, None, None, '2577764') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 301 time used for this insertion : 0.036982059478759766 save_final save missing photos in datou_result : time spend for datou_step_exec : 52.63631081581116 time spend to save output : 0.03982234001159668 total time spend for step 2 : 52.676133155822754 step3:thcl Tue Feb 11 21:14:52 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.019934654235839844 time to convert the images to numpy array : 0.6312763690948486 time to import caffe and check if the image exist : 0.011552810668945312 time to convert the images to numpy array : 0.6960046291351318 time to import caffe and check if the image exist : 0.017182111740112305 time to convert the images to numpy array : 0.7241413593292236 time to import caffe and check if the image exist : 0.0192873477935791 time to convert the images to numpy array : 0.8508813381195068 time to import caffe and check if the image exist : 0.023134231567382812 time to convert the images to numpy array : 0.8736536502838135 time to import caffe and check if the image exist : 0.01877307891845703 time to convert the images to numpy array : 0.8813610076904297 time to import caffe and check if the image exist : 0.02080702781677246 time to convert the images to numpy array : 0.8791170120239258 time to import caffe and check if the image exist : 0.014595508575439453 time to convert the images to numpy array : 0.8870279788970947 time to import caffe and check if the image exist : 0.015192747116088867 time to convert the images to numpy array : 0.89263916015625 time to import caffe and check if the image exist : 0.019341707229614258 time to convert the images to numpy array : 1.166719675064087 total time to convert the images to numpy array : 1.9612901210784912 list photo_ids error: [] list photo_ids correct : [-3670624561, -3670624579, -3670624687, -3670624695, -3670624699, -3670624716, -3670624609, -3670624624, -3670624638, -3670624657, -3670624560, -3670624571, -3670624586, -3670624600, -3670624591, -3670624590, -3670624622, -3670624635, -3670624691, -3670624689, -3670624572, -3670624576, -3670624569, -3670624589, -3670624582, -3670624588, -3670624602, -3670624603, -3670624599, -3670624605, -3670624708, -3670624718, -3670624711, -3670624568, -3670624563, -3670624578, -3670624580, -3670624634, -3670624652, -3670624649, -3670624678, -3670624669, -3670624698, -3670624702, -3670624688, -3670624693, -3670624697, -3670624721, -3670624720, -3670624715, -3670624613, -3670624615, -3670624641, -3670624665, -3670624722, -3670624573, -3670624574, -3670624570, -3670624575, -3670624577, -3670624647, -3670624660, -3670624655, -3670624667, -3670624692, -3670624710, -3670624709, -3670624606, -3670624598, -3670624594, -3670624608, -3670624618, -3670624617, -3670624627, -3670624639, -3670624637, -3670624630, -3670624682, -3670624684, -3670624671, -3670624704, -3670624696, -3670624701, -3670624723, -3670624724, -3670624725, -3670624610, -3670624644, -3670624645, -3670624659, -3670624658, -3670624656, -3670624661, -3670624675, -3670624681, -3670624676, -3670624685] number of photos to traite : 97 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5043 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 4824 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.4038243293762207 time used to do the prediction : 0.44414472579956055 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 4822 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 4822 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.543701410293579 time used to do the prediction : 0.17761492729187012 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 1.7404556274414062e-05 save missing photos in datou_result : time spend for datou_step_exec : 13.126756429672241 time spend to save output : 0.0007326602935791016 total time spend for step 3 : 13.12748908996582 step4:merge_mask_thcl_custom Tue Feb 11 21:15:05 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 166 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1336885647, 1336885641, 1336885604, 1336885601, 1336885562, 1336885512, 1336885502, 1336885496, 1336885135, 1336884924] Looping around the photos to save general results len do output : 10 /1336885647Didn't retrieve data .Didn't retrieve data . /1336885641Didn't retrieve data .Didn't retrieve data . /1336885604Didn't retrieve data .Didn't retrieve data . /1336885601Didn't retrieve data .Didn't retrieve data . /1336885562Didn't retrieve data .Didn't retrieve data . /1336885512Didn't retrieve data .Didn't retrieve data . /1336885502Didn't retrieve data .Didn't retrieve data . /1336885496Didn't retrieve data .Didn't retrieve data . /1336885135Didn't retrieve data .Didn't retrieve data . /1336884924Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885647', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885641', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885604', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885601', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885562', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885512', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885502', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885496', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885135', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336884924', None, None, None, None, None, '2577764') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.01238393783569336 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.06737160682678223 time spend to save output : 0.012900114059448242 total time spend for step 4 : 0.08027172088623047 step5:rle_unique_nms_with_priority Tue Feb 11 21:15:05 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 19 nb_hashtags : 7 time to prepare the origin masks : 11.956865549087524 time for calcul the mask position with numpy : 0.21329236030578613 nb_pixel_total : 5920486 time to create 1 rle with new method : 0.6156144142150879 time for calcul the mask position with numpy : 0.033518314361572266 nb_pixel_total : 26145 time to create 1 rle with old method : 0.0280759334564209 time for calcul the mask position with numpy : 0.03348827362060547 nb_pixel_total : 169 time to create 1 rle with old method : 0.0003657341003417969 time for calcul the mask position with numpy : 0.03683114051818848 nb_pixel_total : 86943 time to create 1 rle with old method : 0.09517288208007812 time for calcul the mask position with numpy : 0.040317535400390625 nb_pixel_total : 193626 time to create 1 rle with new method : 0.47263097763061523 time for calcul the mask position with numpy : 0.037197113037109375 nb_pixel_total : 73527 time to create 1 rle with old method : 0.0802159309387207 time for calcul the mask position with numpy : 0.038428306579589844 nb_pixel_total : 182758 time to create 1 rle with new method : 0.35186338424682617 time for calcul the mask position with numpy : 0.03636980056762695 nb_pixel_total : 45271 time to create 1 rle with old method : 0.04848933219909668 time for calcul the mask position with numpy : 0.03292489051818848 nb_pixel_total : 17839 time to create 1 rle with old method : 0.022194385528564453 time for calcul the mask position with numpy : 0.03434562683105469 nb_pixel_total : 7198 time to create 1 rle with old method : 0.008725881576538086 time for calcul the mask position with numpy : 0.03774142265319824 nb_pixel_total : 62609 time to create 1 rle with old method : 0.07248663902282715 time for calcul the mask position with numpy : 0.0371394157409668 nb_pixel_total : 32965 time to create 1 rle with old method : 0.03917574882507324 time for calcul the mask position with numpy : 0.024512767791748047 nb_pixel_total : 3187 time to create 1 rle with old method : 0.0036988258361816406 time for calcul the mask position with numpy : 0.022589683532714844 nb_pixel_total : 7807 time to create 1 rle with old method : 0.009992361068725586 time for calcul the mask position with numpy : 0.026686429977416992 nb_pixel_total : 30417 time to create 1 rle with old method : 0.03349447250366211 time for calcul the mask position with numpy : 0.021550416946411133 nb_pixel_total : 8129 time to create 1 rle with old method : 0.009183406829833984 time for calcul the mask position with numpy : 0.021378517150878906 nb_pixel_total : 1795 time to create 1 rle with old method : 0.0020494461059570312 time for calcul the mask position with numpy : 0.034540414810180664 nb_pixel_total : 173422 time to create 1 rle with new method : 0.8669619560241699 time for calcul the mask position with numpy : 0.035553693771362305 nb_pixel_total : 34090 time to create 1 rle with old method : 0.0380706787109375 time for calcul the mask position with numpy : 0.035069942474365234 nb_pixel_total : 141857 time to create 1 rle with old method : 0.17467331886291504 create new chi : 3.9148166179656982 time to delete rle : 0.015070199966430664 batch 1 Loaded 20 chid ids of type : 3726 Number RLEs to save : 12250 TO DO : save crop sub photo not yet done ! save time : 0.7097961902618408 nb_obj : 11 nb_hashtags : 5 time to prepare the origin masks : 9.479719638824463 time for calcul the mask position with numpy : 0.9516112804412842 nb_pixel_total : 6380020 time to create 1 rle with new method : 0.4174661636352539 time for calcul the mask position with numpy : 0.03339648246765137 nb_pixel_total : 70817 time to create 1 rle with old method : 0.0770723819732666 time for calcul the mask position with numpy : 0.04088282585144043 nb_pixel_total : 342541 time to create 1 rle with new method : 0.6252708435058594 time for calcul the mask position with numpy : 0.07927584648132324 nb_pixel_total : 155024 time to create 1 rle with new method : 0.8232181072235107 time for calcul the mask position with numpy : 0.03740215301513672 nb_pixel_total : 13088 time to create 1 rle with old method : 0.014988422393798828 time for calcul the mask position with numpy : 0.03594040870666504 nb_pixel_total : 2084 time to create 1 rle with old method : 0.00246429443359375 time for calcul the mask position with numpy : 0.03615593910217285 nb_pixel_total : 2652 time to create 1 rle with old method : 0.003173351287841797 time for calcul the mask position with numpy : 0.030004262924194336 nb_pixel_total : 17185 time to create 1 rle with old method : 0.028474807739257812 time for calcul the mask position with numpy : 0.02750420570373535 nb_pixel_total : 47690 time to create 1 rle with old method : 0.06142020225524902 time for calcul the mask position with numpy : 0.029925823211669922 nb_pixel_total : 4024 time to create 1 rle with old method : 0.004746913909912109 time for calcul the mask position with numpy : 0.03407907485961914 nb_pixel_total : 12082 time to create 1 rle with old method : 0.01350259780883789 time for calcul the mask position with numpy : 0.04962897300720215 nb_pixel_total : 3033 time to create 1 rle with old method : 0.005147695541381836 create new chi : 3.5614988803863525 time to delete rle : 0.0015223026275634766 batch 1 Loaded 12 chid ids of type : 3726 Number RLEs to save : 8074 TO DO : save crop sub photo not yet done ! save time : 0.48510074615478516 nb_obj : 24 nb_hashtags : 6 time to prepare the origin masks : 13.206918001174927 time for calcul the mask position with numpy : 0.643092155456543 nb_pixel_total : 5898547 time to create 1 rle with new method : 0.8381664752960205 time for calcul the mask position with numpy : 0.025730371475219727 nb_pixel_total : 4166 time to create 1 rle with old method : 0.005147218704223633 time for calcul the mask position with numpy : 0.04342031478881836 nb_pixel_total : 53222 time to create 1 rle with old method : 0.057791709899902344 time for calcul the mask position with numpy : 0.04070329666137695 nb_pixel_total : 1804 time to create 1 rle with old method : 0.0021522045135498047 time for calcul the mask position with numpy : 0.04024815559387207 nb_pixel_total : 1871 time to create 1 rle with old method : 0.002197742462158203 time for calcul the mask position with numpy : 0.04065299034118652 nb_pixel_total : 102040 time to create 1 rle with old method : 0.10810685157775879 time for calcul the mask position with numpy : 0.044938087463378906 nb_pixel_total : 354869 time to create 1 rle with new method : 0.3575558662414551 time for calcul the mask position with numpy : 0.04919600486755371 nb_pixel_total : 22506 time to create 1 rle with old method : 0.024082660675048828 time for calcul the mask position with numpy : 0.040010690689086914 nb_pixel_total : 4087 time to create 1 rle with old method : 0.0047550201416015625 time for calcul the mask position with numpy : 0.04039740562438965 nb_pixel_total : 2888 time to create 1 rle with old method : 0.0035576820373535156 time for calcul the mask position with numpy : 0.039704084396362305 nb_pixel_total : 9098 time to create 1 rle with old method : 0.027608633041381836 time for calcul the mask position with numpy : 0.04076051712036133 nb_pixel_total : 43962 time to create 1 rle with old method : 0.04679512977600098 time for calcul the mask position with numpy : 0.043314218521118164 nb_pixel_total : 35084 time to create 1 rle with old method : 0.0384211540222168 time for calcul the mask position with numpy : 0.04253554344177246 nb_pixel_total : 12887 time to create 1 rle with old method : 0.020441293716430664 time for calcul the mask position with numpy : 0.043076276779174805 nb_pixel_total : 7220 time to create 1 rle with old method : 0.008193492889404297 time for calcul the mask position with numpy : 0.041990041732788086 nb_pixel_total : 3766 time to create 1 rle with old method : 0.00426030158996582 time for calcul the mask position with numpy : 0.04351162910461426 nb_pixel_total : 38763 time to create 1 rle with old method : 0.041886091232299805 time for calcul the mask position with numpy : 0.040524959564208984 nb_pixel_total : 39412 time to create 1 rle with old method : 0.043810129165649414 time for calcul the mask position with numpy : 0.0412142276763916 nb_pixel_total : 47167 time to create 1 rle with old method : 0.050923824310302734 time for calcul the mask position with numpy : 0.04727888107299805 nb_pixel_total : 11857 time to create 1 rle with old method : 0.016802310943603516 time for calcul the mask position with numpy : 0.04187965393066406 nb_pixel_total : 11021 time to create 1 rle with old method : 0.012375831604003906 time for calcul the mask position with numpy : 0.03694629669189453 nb_pixel_total : 236022 time to create 1 rle with new method : 1.0759658813476562 time for calcul the mask position with numpy : 0.030143022537231445 nb_pixel_total : 94823 time to create 1 rle with old method : 0.10720682144165039 time for calcul the mask position with numpy : 0.04271078109741211 nb_pixel_total : 5795 time to create 1 rle with old method : 0.006341457366943359 time for calcul the mask position with numpy : 0.043607473373413086 nb_pixel_total : 7363 time to create 1 rle with old method : 0.00818777084350586 create new chi : 4.661108732223511 time to delete rle : 0.0019059181213378906 batch 1 Loaded 25 chid ids of type : 3726 Number RLEs to save : 13950 TO DO : save crop sub photo not yet done ! save time : 0.7822673320770264 nb_obj : 13 nb_hashtags : 3 time to prepare the origin masks : 11.047541618347168 time for calcul the mask position with numpy : 0.46157050132751465 nb_pixel_total : 5815577 time to create 1 rle with new method : 0.7561953067779541 time for calcul the mask position with numpy : 0.02168893814086914 nb_pixel_total : 22561 time to create 1 rle with old method : 0.024524211883544922 time for calcul the mask position with numpy : 0.024633407592773438 nb_pixel_total : 151291 time to create 1 rle with new method : 0.6822435855865479 time for calcul the mask position with numpy : 0.022937774658203125 nb_pixel_total : 5423 time to create 1 rle with old method : 0.0062448978424072266 time for calcul the mask position with numpy : 0.02220010757446289 nb_pixel_total : 11129 time to create 1 rle with old method : 0.01389765739440918 time for calcul the mask position with numpy : 0.06218242645263672 nb_pixel_total : 590641 time to create 1 rle with new method : 0.8019719123840332 time for calcul the mask position with numpy : 0.021632671356201172 nb_pixel_total : 731 time to create 1 rle with old method : 0.0009007453918457031 time for calcul the mask position with numpy : 0.022656679153442383 nb_pixel_total : 158665 time to create 1 rle with new method : 0.5617032051086426 time for calcul the mask position with numpy : 0.035765886306762695 nb_pixel_total : 17759 time to create 1 rle with old method : 0.02288818359375 time for calcul the mask position with numpy : 0.03249859809875488 nb_pixel_total : 34216 time to create 1 rle with old method : 0.04338860511779785 time for calcul the mask position with numpy : 0.027881622314453125 nb_pixel_total : 144838 time to create 1 rle with old method : 0.15299463272094727 time for calcul the mask position with numpy : 0.021259784698486328 nb_pixel_total : 10496 time to create 1 rle with old method : 0.011487960815429688 time for calcul the mask position with numpy : 0.021389245986938477 nb_pixel_total : 52970 time to create 1 rle with old method : 0.05501151084899902 time for calcul the mask position with numpy : 0.021430015563964844 nb_pixel_total : 33943 time to create 1 rle with old method : 0.03609633445739746 create new chi : 4.112091779708862 time to delete rle : 0.0012450218200683594 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 12790 TO DO : save crop sub photo not yet done ! save time : 0.7152810096740723 nb_obj : 12 nb_hashtags : 4 time to prepare the origin masks : 8.692389726638794 time for calcul the mask position with numpy : 0.5981287956237793 nb_pixel_total : 6502642 time to create 1 rle with new method : 0.550560474395752 time for calcul the mask position with numpy : 0.02243638038635254 nb_pixel_total : 40553 time to create 1 rle with old method : 0.04683852195739746 time for calcul the mask position with numpy : 0.023320436477661133 nb_pixel_total : 2007 time to create 1 rle with old method : 0.002542734146118164 time for calcul the mask position with numpy : 0.023871660232543945 nb_pixel_total : 41111 time to create 1 rle with old method : 0.04663348197937012 time for calcul the mask position with numpy : 0.02352142333984375 nb_pixel_total : 93887 time to create 1 rle with old method : 0.10038542747497559 time for calcul the mask position with numpy : 0.021843910217285156 nb_pixel_total : 9677 time to create 1 rle with old method : 0.01051640510559082 time for calcul the mask position with numpy : 0.0224912166595459 nb_pixel_total : 3975 time to create 1 rle with old method : 0.005425930023193359 time for calcul the mask position with numpy : 0.02401590347290039 nb_pixel_total : 24513 time to create 1 rle with old method : 0.028662681579589844 time for calcul the mask position with numpy : 0.023576736450195312 nb_pixel_total : 6209 time to create 1 rle with old method : 0.007328510284423828 time for calcul the mask position with numpy : 0.025731563568115234 nb_pixel_total : 154219 time to create 1 rle with new method : 0.636829137802124 time for calcul the mask position with numpy : 0.021603107452392578 nb_pixel_total : 8996 time to create 1 rle with old method : 0.010180234909057617 time for calcul the mask position with numpy : 0.022477388381958008 nb_pixel_total : 123589 time to create 1 rle with old method : 0.13769078254699707 time for calcul the mask position with numpy : 0.022548913955688477 nb_pixel_total : 38862 time to create 1 rle with old method : 0.043691158294677734 create new chi : 2.5742764472961426 time to delete rle : 0.0008959770202636719 batch 1 Loaded 13 chid ids of type : 3726 Number RLEs to save : 8675 TO DO : save crop sub photo not yet done ! save time : 0.6774570941925049 nb_obj : 20 nb_hashtags : 6 time to prepare the origin masks : 14.24393630027771 time for calcul the mask position with numpy : 0.47722721099853516 nb_pixel_total : 5553550 time to create 1 rle with new method : 1.0264413356781006 time for calcul the mask position with numpy : 0.023602962493896484 nb_pixel_total : 155210 time to create 1 rle with new method : 0.42415404319763184 time for calcul the mask position with numpy : 0.023644685745239258 nb_pixel_total : 14922 time to create 1 rle with old method : 0.01803755760192871 time for calcul the mask position with numpy : 0.024915695190429688 nb_pixel_total : 10357 time to create 1 rle with old method : 0.013353824615478516 time for calcul the mask position with numpy : 0.026154041290283203 nb_pixel_total : 160085 time to create 1 rle with new method : 0.6823360919952393 time for calcul the mask position with numpy : 0.027480602264404297 nb_pixel_total : 116009 time to create 1 rle with old method : 0.13105440139770508 time for calcul the mask position with numpy : 0.029540061950683594 nb_pixel_total : 189260 time to create 1 rle with new method : 0.7082347869873047 time for calcul the mask position with numpy : 0.02712416648864746 nb_pixel_total : 17088 time to create 1 rle with old method : 0.01915717124938965 time for calcul the mask position with numpy : 0.0294187068939209 nb_pixel_total : 831 time to create 1 rle with old method : 0.0010683536529541016 time for calcul the mask position with numpy : 0.029432058334350586 nb_pixel_total : 94244 time to create 1 rle with old method : 0.1025238037109375 time for calcul the mask position with numpy : 0.027395009994506836 nb_pixel_total : 25147 time to create 1 rle with old method : 0.0271608829498291 time for calcul the mask position with numpy : 0.028105974197387695 nb_pixel_total : 308702 time to create 1 rle with new method : 0.548382043838501 time for calcul the mask position with numpy : 0.02780604362487793 nb_pixel_total : 2258 time to create 1 rle with old method : 0.002642393112182617 time for calcul the mask position with numpy : 0.026484251022338867 nb_pixel_total : 30651 time to create 1 rle with old method : 0.033719539642333984 time for calcul the mask position with numpy : 0.027318239212036133 nb_pixel_total : 11687 time to create 1 rle with old method : 0.013630390167236328 time for calcul the mask position with numpy : 0.025696516036987305 nb_pixel_total : 4802 time to create 1 rle with old method : 0.0055217742919921875 time for calcul the mask position with numpy : 0.027722835540771484 nb_pixel_total : 87742 time to create 1 rle with old method : 0.09323310852050781 time for calcul the mask position with numpy : 0.026871204376220703 nb_pixel_total : 202814 time to create 1 rle with new method : 0.5324885845184326 time for calcul the mask position with numpy : 0.030514240264892578 nb_pixel_total : 40 time to create 1 rle with old method : 0.0003299713134765625 time for calcul the mask position with numpy : 0.02636551856994629 nb_pixel_total : 45921 time to create 1 rle with old method : 0.05173373222351074 time for calcul the mask position with numpy : 0.030562639236450195 nb_pixel_total : 18920 time to create 1 rle with old method : 0.0209195613861084 create new chi : 5.684443235397339 time to delete rle : 0.0017158985137939453 batch 1 Loaded 21 chid ids of type : 3726 Number RLEs to save : 14513 TO DO : save crop sub photo not yet done ! save time : 0.9594042301177979 nb_obj : 21 nb_hashtags : 7 time to prepare the origin masks : 14.29454755783081 time for calcul the mask position with numpy : 0.3886733055114746 nb_pixel_total : 5341819 time to create 1 rle with new method : 1.0444767475128174 time for calcul the mask position with numpy : 0.037180423736572266 nb_pixel_total : 1287 time to create 1 rle with old method : 0.0016579627990722656 time for calcul the mask position with numpy : 0.03774738311767578 nb_pixel_total : 65862 time to create 1 rle with old method : 0.08051443099975586 time for calcul the mask position with numpy : 0.0460050106048584 nb_pixel_total : 85105 time to create 1 rle with old method : 0.10009241104125977 time for calcul the mask position with numpy : 0.04207420349121094 nb_pixel_total : 2861 time to create 1 rle with old method : 0.003381013870239258 time for calcul the mask position with numpy : 0.036608219146728516 nb_pixel_total : 157741 time to create 1 rle with new method : 0.6281812191009521 time for calcul the mask position with numpy : 0.038823843002319336 nb_pixel_total : 11305 time to create 1 rle with old method : 0.013161182403564453 time for calcul the mask position with numpy : 0.041536569595336914 nb_pixel_total : 4188 time to create 1 rle with old method : 0.0057146549224853516 time for calcul the mask position with numpy : 0.04698300361633301 nb_pixel_total : 3831 time to create 1 rle with old method : 0.00594639778137207 time for calcul the mask position with numpy : 0.05755901336669922 nb_pixel_total : 715746 time to create 1 rle with new method : 1.2343125343322754 time for calcul the mask position with numpy : 0.04742264747619629 nb_pixel_total : 3663 time to create 1 rle with old method : 0.004490375518798828 time for calcul the mask position with numpy : 0.04757380485534668 nb_pixel_total : 166554 time to create 1 rle with new method : 0.7795755863189697 time for calcul the mask position with numpy : 0.028486251831054688 nb_pixel_total : 105214 time to create 1 rle with old method : 0.1199798583984375 time for calcul the mask position with numpy : 0.03653740882873535 nb_pixel_total : 28087 time to create 1 rle with old method : 0.03157353401184082 time for calcul the mask position with numpy : 0.03349781036376953 nb_pixel_total : 2363 time to create 1 rle with old method : 0.002722501754760742 time for calcul the mask position with numpy : 0.03414177894592285 nb_pixel_total : 43242 time to create 1 rle with old method : 0.047301292419433594 time for calcul the mask position with numpy : 0.039522409439086914 nb_pixel_total : 20205 time to create 1 rle with old method : 0.022996902465820312 time for calcul the mask position with numpy : 0.03716278076171875 nb_pixel_total : 25539 time to create 1 rle with old method : 0.02875661849975586 time for calcul the mask position with numpy : 0.04334235191345215 nb_pixel_total : 1867 time to create 1 rle with old method : 0.002477407455444336 time for calcul the mask position with numpy : 0.023749113082885742 nb_pixel_total : 91167 time to create 1 rle with old method : 0.12682080268859863 time for calcul the mask position with numpy : 0.02369856834411621 nb_pixel_total : 125160 time to create 1 rle with old method : 0.17311668395996094 time for calcul the mask position with numpy : 0.022727251052856445 nb_pixel_total : 47434 time to create 1 rle with old method : 0.05086851119995117 create new chi : 5.847565650939941 time to delete rle : 0.002275228500366211 batch 1 Loaded 22 chid ids of type : 3726 Number RLEs to save : 15434 TO DO : save crop sub photo not yet done ! save time : 0.855313777923584 nb_obj : 18 nb_hashtags : 6 time to prepare the origin masks : 12.643817663192749 time for calcul the mask position with numpy : 0.6920232772827148 nb_pixel_total : 5946673 time to create 1 rle with new method : 0.9288067817687988 time for calcul the mask position with numpy : 0.04824519157409668 nb_pixel_total : 3206 time to create 1 rle with old method : 0.00452733039855957 time for calcul the mask position with numpy : 0.049314260482788086 nb_pixel_total : 29423 time to create 1 rle with old method : 0.03825712203979492 time for calcul the mask position with numpy : 0.026805639266967773 nb_pixel_total : 3512 time to create 1 rle with old method : 0.003975391387939453 time for calcul the mask position with numpy : 0.03201103210449219 nb_pixel_total : 152110 time to create 1 rle with new method : 1.8526325225830078 time for calcul the mask position with numpy : 0.03335738182067871 nb_pixel_total : 5269 time to create 1 rle with old method : 0.009801387786865234 time for calcul the mask position with numpy : 0.03260231018066406 nb_pixel_total : 368947 time to create 1 rle with new method : 0.484363317489624 time for calcul the mask position with numpy : 0.03174471855163574 nb_pixel_total : 19978 time to create 1 rle with old method : 0.022832870483398438 time for calcul the mask position with numpy : 0.030720233917236328 nb_pixel_total : 21982 time to create 1 rle with old method : 0.02728128433227539 time for calcul the mask position with numpy : 0.028443336486816406 nb_pixel_total : 3703 time to create 1 rle with old method : 0.00444340705871582 time for calcul the mask position with numpy : 0.031445980072021484 nb_pixel_total : 39865 time to create 1 rle with old method : 0.05593419075012207 time for calcul the mask position with numpy : 0.037152767181396484 nb_pixel_total : 104035 time to create 1 rle with old method : 0.1308450698852539 time for calcul the mask position with numpy : 0.033173561096191406 nb_pixel_total : 8661 time to create 1 rle with old method : 0.012119531631469727 time for calcul the mask position with numpy : 0.033779144287109375 nb_pixel_total : 154235 time to create 1 rle with new method : 0.8171055316925049 time for calcul the mask position with numpy : 0.02403712272644043 nb_pixel_total : 10747 time to create 1 rle with old method : 0.020836353302001953 time for calcul the mask position with numpy : 0.024528980255126953 nb_pixel_total : 7187 time to create 1 rle with old method : 0.008450508117675781 time for calcul the mask position with numpy : 0.02785015106201172 nb_pixel_total : 99506 time to create 1 rle with old method : 0.10906863212585449 time for calcul the mask position with numpy : 0.02606987953186035 nb_pixel_total : 8030 time to create 1 rle with old method : 0.010200262069702148 time for calcul the mask position with numpy : 0.026441335678100586 nb_pixel_total : 63171 time to create 1 rle with old method : 0.07144331932067871 create new chi : 6.035593271255493 time to delete rle : 0.0013980865478515625 batch 1 Loaded 19 chid ids of type : 3726 Number RLEs to save : 12507 TO DO : save crop sub photo not yet done ! save time : 0.7343094348907471 nb_obj : 3 nb_hashtags : 3 time to prepare the origin masks : 4.433246374130249 time for calcul the mask position with numpy : 0.04190421104431152 nb_pixel_total : 2115675 time to create 1 rle with new method : 0.9551413059234619 time for calcul the mask position with numpy : 0.024791955947875977 nb_pixel_total : 127696 time to create 1 rle with old method : 0.1425619125366211 time for calcul the mask position with numpy : 0.02328658103942871 nb_pixel_total : 7699 time to create 1 rle with old method : 0.008461952209472656 time for calcul the mask position with numpy : 0.7402544021606445 nb_pixel_total : 4799170 time to create 1 rle with new method : 0.6137425899505615 create new chi : 2.6169559955596924 time to delete rle : 0.0010411739349365234 batch 1 Loaded 4 chid ids of type : 3726 Number RLEs to save : 9842 TO DO : save crop sub photo not yet done ! save time : 0.5822110176086426 nb_obj : 15 nb_hashtags : 5 time to prepare the origin masks : 9.131714820861816 time for calcul the mask position with numpy : 0.32525062561035156 nb_pixel_total : 3996196 time to create 1 rle with new method : 0.39630579948425293 time for calcul the mask position with numpy : 0.026613473892211914 nb_pixel_total : 4222 time to create 1 rle with old method : 0.0051043033599853516 time for calcul the mask position with numpy : 0.0256500244140625 nb_pixel_total : 61609 time to create 1 rle with old method : 0.0797417163848877 time for calcul the mask position with numpy : 0.027265548706054688 nb_pixel_total : 38131 time to create 1 rle with old method : 0.04824662208557129 time for calcul the mask position with numpy : 0.026009559631347656 nb_pixel_total : 17524 time to create 1 rle with old method : 0.018913984298706055 time for calcul the mask position with numpy : 0.026743650436401367 nb_pixel_total : 40930 time to create 1 rle with old method : 0.045908212661743164 time for calcul the mask position with numpy : 0.025411128997802734 nb_pixel_total : 4102 time to create 1 rle with old method : 0.004823207855224609 time for calcul the mask position with numpy : 0.06935787200927734 nb_pixel_total : 149156 time to create 1 rle with old method : 0.16232705116271973 time for calcul the mask position with numpy : 0.029758453369140625 nb_pixel_total : 430473 time to create 1 rle with new method : 2.8889219760894775 time for calcul the mask position with numpy : 0.026474714279174805 nb_pixel_total : 98672 time to create 1 rle with old method : 0.10622310638427734 time for calcul the mask position with numpy : 0.05428600311279297 nb_pixel_total : 1960226 time to create 1 rle with new method : 0.9426159858703613 time for calcul the mask position with numpy : 0.024892091751098633 nb_pixel_total : 5135 time to create 1 rle with old method : 0.00527644157409668 time for calcul the mask position with numpy : 0.0260927677154541 nb_pixel_total : 8670 time to create 1 rle with old method : 0.00994110107421875 time for calcul the mask position with numpy : 0.027134180068969727 nb_pixel_total : 71285 time to create 1 rle with old method : 0.07357239723205566 time for calcul the mask position with numpy : 0.025707006454467773 nb_pixel_total : 104290 time to create 1 rle with old method : 0.1057896614074707 time for calcul the mask position with numpy : 0.0264284610748291 nb_pixel_total : 59619 time to create 1 rle with old method : 0.061338186264038086 create new chi : 5.853396892547607 time to delete rle : 0.0019044876098632812 batch 1 Loaded 16 chid ids of type : 3726 Number RLEs to save : 18019 TO DO : save crop sub photo not yet done ! save time : 0.9944906234741211 map_output_result : {1336885647: (0.0, 'Should be the crop_list due to order', 0.0), 1336885641: (0.0, 'Should be the crop_list due to order', 0.0), 1336885604: (0.0, 'Should be the crop_list due to order', 0.0), 1336885601: (0.0, 'Should be the crop_list due to order', 0.0), 1336885562: (0.0, 'Should be the crop_list due to order', 0.0), 1336885512: (0.0, 'Should be the crop_list due to order', 0.0), 1336885502: (0.0, 'Should be the crop_list due to order', 0.0), 1336885496: (0.0, 'Should be the crop_list due to order', 0.0), 1336885135: (0.0, 'Should be the crop_list due to order', 0.0), 1336884924: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1336885647, 1336885641, 1336885604, 1336885601, 1336885562, 1336885512, 1336885502, 1336885496, 1336885135, 1336884924] Looping around the photos to save general results len do output : 10 /1336885647.Didn't retrieve data . /1336885641.Didn't retrieve data . /1336885604.Didn't retrieve data . /1336885601.Didn't retrieve data . /1336885562.Didn't retrieve data . /1336885512.Didn't retrieve data . /1336885502.Didn't retrieve data . /1336885496.Didn't retrieve data . /1336885135.Didn't retrieve data . /1336884924.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885647', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885641', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885604', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885601', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885562', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885512', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885502', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885496', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885135', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336884924', None, None, None, None, None, '2577764') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013100862503051758 save_final save missing photos in datou_result : time spend for datou_step_exec : 162.52244997024536 time spend to save output : 0.02610611915588379 total time spend for step 5 : 162.54855608940125 step6:crop_condition Tue Feb 11 21:17:48 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 166 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739305071_7911 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.5030128955841064 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! About to upload 9 photos upload in portfolio : 4869462 init cache_photo without model_param we have 9 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739305083_7911 we have uploaded 9 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.9847686290740967 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739305089_7911 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.325329065322876 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! About to upload 5 photos upload in portfolio : 4869462 init cache_photo without model_param we have 5 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739305094_7911 we have uploaded 5 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.4824137687683105 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! About to upload 5 photos upload in portfolio : 4869462 init cache_photo without model_param we have 5 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739305098_7911 we have uploaded 5 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.3821079730987549 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! About to upload 5 photos upload in portfolio : 4869462 init cache_photo without model_param we have 5 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739305103_7911 we have uploaded 5 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.456646203994751 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739305106_7911 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.5826816558837891 we have finished the crop for the class : metal delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1336885647, 1336885641, 1336885604, 1336885601, 1336885562, 1336885512, 1336885502, 1336885496, 1336885135, 1336884924] Looping around the photos to save general results len do output : 33 /1336909704Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909705Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909706Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909707Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909723Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909724Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909725Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909726Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909727Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909728Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909729Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909730Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909731Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909732Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909733Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909734Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909735Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909737Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909738Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909739Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909740Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909741Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909743Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909744Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909745Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909746Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909747Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909749Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909750Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909751Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909752Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909753Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1336909755Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885647', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885641', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885604', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885601', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885562', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885512', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885502', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885496', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885135', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336884924', None, None, None, None, None, '2577764') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 109 time used for this insertion : 0.03255820274353027 save_final save missing photos in datou_result : time spend for datou_step_exec : 39.81237196922302 time spend to save output : 0.03360748291015625 total time spend for step 6 : 39.84597945213318 step7:ventilate_hashtags_in_portfolio Tue Feb 11 21:18:28 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20461712 get user id for portfolio 20461712 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20461712 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('metal','flou','plastique','Teint_Dans_La_Masse','Carton_brun','mal_croppe','papier','autre_refus','cartonnette','kraft','Carton_gris','environnement')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20461712 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('metal','flou','plastique','Teint_Dans_La_Masse','Carton_brun','mal_croppe','papier','autre_refus','cartonnette','kraft','Carton_gris','environnement')) AND mptpi.`min_score`=0.5 To do To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20461712 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('metal','flou','plastique','Teint_Dans_La_Masse','Carton_brun','mal_croppe','papier','autre_refus','cartonnette','kraft','Carton_gris','environnement')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20461867,20461868,20461869,20461870,20461871,20461872,20461873,20461874,20461875,20461876,20461877,20461878?tags=metal,flou,plastique,Teint_Dans_La_Masse,Carton_brun,mal_croppe,papier,autre_refus,cartonnette,kraft,Carton_gris,environnement Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1336885647, 1336885641, 1336885604, 1336885601, 1336885562, 1336885512, 1336885502, 1336885496, 1336885135, 1336884924] Looping around the photos to save general results len do output : 1 /20461712. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885647', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885641', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885604', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885601', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885562', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885512', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885502', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885496', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885135', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336884924', None, None, None, None, None, '2577764') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.012864112854003906 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.607064962387085 time spend to save output : 0.013029098510742188 total time spend for step 7 : 1.6200940608978271 step8:final Tue Feb 11 21:18:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1336885647: ('0.11839248806077622',), 1336885641: ('0.11839248806077622',), 1336885604: ('0.11839248806077622',), 1336885601: ('0.11839248806077622',), 1336885562: ('0.11839248806077622',), 1336885512: ('0.11839248806077622',), 1336885502: ('0.11839248806077622',), 1336885496: ('0.11839248806077622',), 1336885135: ('0.11839248806077622',), 1336884924: ('0.11839248806077622',)} new output for save of step final : {1336885647: ('0.11839248806077622',), 1336885641: ('0.11839248806077622',), 1336885604: ('0.11839248806077622',), 1336885601: ('0.11839248806077622',), 1336885562: ('0.11839248806077622',), 1336885512: ('0.11839248806077622',), 1336885502: ('0.11839248806077622',), 1336885496: ('0.11839248806077622',), 1336885135: ('0.11839248806077622',), 1336884924: ('0.11839248806077622',)} [1336885647, 1336885641, 1336885604, 1336885601, 1336885562, 1336885512, 1336885502, 1336885496, 1336885135, 1336884924] Looping around the photos to save general results len do output : 10 /1336885647.Didn't retrieve data . /1336885641.Didn't retrieve data . /1336885604.Didn't retrieve data . /1336885601.Didn't retrieve data . /1336885562.Didn't retrieve data . /1336885512.Didn't retrieve data . /1336885502.Didn't retrieve data . /1336885496.Didn't retrieve data . /1336885135.Didn't retrieve data . /1336884924.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885647', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885641', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885604', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885601', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885562', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885512', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885502', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885496', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885135', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336884924', None, None, None, None, None, '2577764') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013180732727050781 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.11378788948059082 time spend to save output : 0.014075994491577148 total time spend for step 8 : 0.12786388397216797 step9:velours_tree Tue Feb 11 21:18:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.2875950336456299 time spend to save output : 4.076957702636719e-05 total time spend for step 9 : 0.28763580322265625 step10:send_mail_cod Tue Feb 11 21:18:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20461712_11-02-2025_21_18_30.pdf 20461867 change filename to text .imagette204618671739305110 20461868 imagette204618681739305110 20461869 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette204618691739305110 20461872 imagette204618721739305110 20461874 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette204618741739305110 20461875 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette204618751739305111 20461876 imagette204618761739305111 20461870 change filename to text .change filename to text .change filename to text .change filename to text .imagette204618701739305111 20461871 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette204618711739305111 20461877 change filename to text .change filename to text .change filename to text .change filename to text .imagette204618771739305112 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20461712 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20461867,20461868,20461869,20461870,20461871,20461872,20461873,20461874,20461875,20461876,20461877,20461878?tags=metal,flou,plastique,Teint_Dans_La_Masse,Carton_brun,mal_croppe,papier,autre_refus,cartonnette,kraft,Carton_gris,environnement your option no_mail is active, we will not send the real mail to your client args[1336885647] : ('0.11839248806077622',) no score found for photo 1336885647 apple ('0.11839248806077622',) We are sending mail with results at report@fotonower.com args[1336885641] : ('0.11839248806077622',) no score found for photo 1336885641 apple ('0.11839248806077622',) We are sending mail with results at report@fotonower.com args[1336885604] : ('0.11839248806077622',) no score found for photo 1336885604 apple ('0.11839248806077622',) We are sending mail with results at report@fotonower.com args[1336885601] : ('0.11839248806077622',) no score found for photo 1336885601 apple ('0.11839248806077622',) We are sending mail with results at report@fotonower.com args[1336885562] : ('0.11839248806077622',) no score found for photo 1336885562 apple ('0.11839248806077622',) We are sending mail with results at report@fotonower.com args[1336885512] : ('0.11839248806077622',) no score found for photo 1336885512 apple ('0.11839248806077622',) We are sending mail with results at report@fotonower.com args[1336885502] : ('0.11839248806077622',) no score found for photo 1336885502 apple ('0.11839248806077622',) We are sending mail with results at report@fotonower.com args[1336885496] : ('0.11839248806077622',) no score found for photo 1336885496 apple ('0.11839248806077622',) We are sending mail with results at report@fotonower.com args[1336885135] : ('0.11839248806077622',) no score found for photo 1336885135 apple ('0.11839248806077622',) We are sending mail with results at report@fotonower.com args[1336884924] : ('0.11839248806077622',) no score found for photo 1336884924 apple ('0.11839248806077622',) We are sending mail with results at report@fotonower.com refus_total : 0.11839248806077622 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20461712 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1336867303,1336885647,1336885641,1336885604,1336885562,1336885512,1336885502,1336884922,1336867305,1336866945,1336867300,1336867297,1336867257,1336867251,1336867056,1336867047,1336867045,1336867043,1336867040,1336867053) Found this number of photos: 20 begin to download photo : 1336867303 begin to download photo : 1336885512 begin to download photo : 1336867300 begin to download photo : 1336867047 download finish for photo 1336885512 begin to download photo : 1336885502 download finish for photo 1336867303 begin to download photo : 1336885647 download finish for photo 1336867300 begin to download photo : 1336867297 download finish for photo 1336867047 begin to download photo : 1336867045 download finish for photo 1336885502 begin to download photo : 1336884922 download finish for photo 1336867297 begin to download photo : 1336867257 download finish for photo 1336885647 begin to download photo : 1336885641 download finish for photo 1336867045 begin to download photo : 1336867043 download finish for photo 1336884922 begin to download photo : 1336867305 download finish for photo 1336867257 begin to download photo : 1336867251 download finish for photo 1336885641 begin to download photo : 1336885604 download finish for photo 1336867305 begin to download photo : 1336866945 download finish for photo 1336867043 begin to download photo : 1336867040 download finish for photo 1336885604 begin to download photo : 1336885562 download finish for photo 1336867251 begin to download photo : 1336867056 download finish for photo 1336866945 download finish for photo 1336867040 begin to download photo : 1336867053 download finish for photo 1336867056 download finish for photo 1336885562 download finish for photo 1336867053 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20461712_11-02-2025_21_18_30.pdf results_Auto_P20461712_11-02-2025_21_18_30.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20461712_11-02-2025_21_18_30.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20461712','results_Auto_P20461712_11-02-2025_21_18_30.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20461712_11-02-2025_21_18_30.pdf','pdf','','0.99','0.11839248806077622') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1336885647, 1336885641, 1336885604, 1336885601, 1336885562, 1336885512, 1336885502, 1336885496, 1336885135, 1336884924] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885647', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885641', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885604', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885601', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885562', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885512', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885502', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885496', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885135', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336884924', None, None, None, None, None, '2577764') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.014639854431152344 save_final save missing photos in datou_result : time spend for datou_step_exec : 6.645322799682617 time spend to save output : 0.014762401580810547 total time spend for step 10 : 6.660085201263428 step11:split_time_score Tue Feb 11 21:18:36 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('17', 28),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 11022025 20461712 Nombre de photos uploadées : 28 / 23040 (0%) 11022025 20461712 Nombre de photos taguées (types de déchets): 0 / 28 (0%) 11022025 20461712 Nombre de photos taguées (volume) : 0 / 28 (0%) elapsed_time : load_data_split_time_score 4.291534423828125e-06 elapsed_time : order_list_meta_photo_and_scores 1.2159347534179688e-05 ???????????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0016510486602783203 elapsed_time : insert_dashboard_record_day_entry 0.02188873291015625 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461688 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461689 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461690 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461691 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461692 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461693 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461694 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461695 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461696 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461697 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461698 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461699 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461700 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461701 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461702 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461703 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461704 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461705 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461706 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461707 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461708 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461709 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461710 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461711 order by id desc limit 1 Qualite : 0.11839248806077622 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20461712_11-02-2025_21_18_30.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461712 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20461712 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461713 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461714 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461715 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461716 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461717 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461718 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20461719 order by id desc limit 1 NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'11022025': {'nb_upload': 28, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1336885647, 1336885641, 1336885604, 1336885601, 1336885562, 1336885512, 1336885502, 1336885496, 1336885135, 1336884924] Looping around the photos to save general results len do output : 1 /20461712Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885647', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885641', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885604', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885601', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885562', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885512', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885502', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885496', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336885135', None, None, None, None, None, '2577764') ('3459', None, None, None, None, None, None, None, '2577764') ('3459', '20461712', '1336884924', None, None, None, None, None, '2577764') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.014455318450927734 save_final save missing photos in datou_result : time spend for datou_step_exec : 23.926727533340454 time spend to save output : 0.0146942138671875 total time spend for step 11 : 23.94142174720764 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 191.66user 166.63system 6:37.22elapsed 90%CPU (0avgtext+0avgdata 3547136maxresident)k 2338144inputs+164856outputs (22519major+14272103minor)pagefaults 0swaps