python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' -a 4741 -l 30 ' -s datou_current_4741 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 471565 load datou : 4741 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 13156 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 13160 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 13160 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 13157 crop_condition is not consistent : 4 used against 2 in the step definition ! WARNING : number of outputs for step 13157 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 13161 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 13159 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 13159 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 13156 doesn't seem to be define in the database( WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 13161 doesn't seem to be define in the database( WARNING : type of input 3 of step 13159 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13163 have datatype=6 whereas input 2 of step 13157 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13162 have datatype=6 whereas input 2 of step 13157 have datatype=None We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 13161 have datatype=10 whereas input 3 of step 13164 have datatype=6 WARNING : output 0 of step 13161 have datatype=10 whereas input 0 of step 13166 have datatype=18 WARNING : type of input 5 of step 13164 doesn't seem to be define in the database( WARNING : output 0 of step 13166 have datatype=11 whereas input 5 of step 13164 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 1 of step 13158 have datatype=7 whereas input 2 of step 13157 have datatype=None WARNING : type of output 3 of step 13157 doesn't seem to be define in the database( WARNING : type of input 3 of step 13161 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13163 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13162 doesn't seem to be define in the database( DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5732, 'mask_refus_amiens_050123', 16384, 25088, 'mask_refus_amiens_050123', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 1, 5, 13, 23, 59), datetime.datetime(2023, 1, 5, 13, 23, 59)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5972, 'learn_entrant_syctomXV_111023', 16384, 25088, 'learn_entrant_syctomXV_111023', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 10, 11, 15, 57, 35), datetime.datetime(2023, 10, 11, 15, 57, 35)) load thcls load THCL from format json or kwargs add thcl : 3453 in CacheModelConfig load THCL from format json or kwargs add thcl : 3783 in CacheModelConfig load pdts add pdt : 5732 in CacheModelConfig add pdt : 5972 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 list_input_json: [] Current got : datou_id : 4741, datou_cur_ids : ['2654458'] with mtr_portfolio_ids : ['21222676'] and first list_photo_ids : [] new path : /proc/471565/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 13156 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 13160 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 13160 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 13157 crop_condition is not consistent : 4 used against 2 in the step definition ! WARNING : number of outputs for step 13157 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 13161 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 13159 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 13159 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 13156 doesn't seem to be define in the database( WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 13161 doesn't seem to be define in the database( WARNING : type of input 3 of step 13159 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13163 have datatype=6 whereas input 2 of step 13157 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13162 have datatype=6 whereas input 2 of step 13157 have datatype=None We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 13161 have datatype=10 whereas input 3 of step 13164 have datatype=6 WARNING : output 0 of step 13161 have datatype=10 whereas input 0 of step 13166 have datatype=18 WARNING : type of input 5 of step 13164 doesn't seem to be define in the database( WARNING : output 0 of step 13166 have datatype=11 whereas input 5 of step 13164 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 1 of step 13158 have datatype=7 whereas input 2 of step 13157 have datatype=None WARNING : type of output 3 of step 13157 doesn't seem to be define in the database( WARNING : type of input 3 of step 13161 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13163 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13162 doesn't seem to be define in the database( DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, mask_detect, blur_detection, brightness, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , BFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 4 ; length of list_pids : 4 ; length of list_args : 4 time to download the photos : 4.23784065246582 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Mon Mar 10 20:01:32 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5081 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-03-10 20:01:35.774344: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-10 20:01:35.803069: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-10 20:01:35.805318: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fe214000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-10 20:01:35.805354: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-10 20:01:35.810006: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-10 20:01:35.959070: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x10cc8b70 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-10 20:01:35.959185: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-10 20:01:35.961491: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-10 20:01:35.962374: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-10 20:01:35.967302: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-10 20:01:35.974523: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-10 20:01:35.975627: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-10 20:01:35.991798: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-10 20:01:35.995385: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-10 20:01:36.023085: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-10 20:01:36.024752: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-10 20:01:36.025296: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-10 20:01:36.026139: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-10 20:01:36.026168: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-10 20:01:36.026182: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-10 20:01:36.027874: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4629 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-03-10 20:01:36.468783: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-10 20:01:36.468920: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-10 20:01:36.468946: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-10 20:01:36.468972: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-10 20:01:36.468997: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-10 20:01:36.469017: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-10 20:01:36.469041: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-10 20:01:36.469063: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-10 20:01:36.470298: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-10 20:01:36.471911: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-10 20:01:36.471967: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-10 20:01:36.471986: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-10 20:01:36.472004: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-10 20:01:36.472037: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-10 20:01:36.472055: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-10 20:01:36.472072: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-10 20:01:36.472090: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-10 20:01:36.473074: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-10 20:01:36.473115: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-10 20:01:36.473123: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-10 20:01:36.473131: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-10 20:01:36.474158: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4629 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl3783 thcls : [{'id': 3783, 'mtr_user_id': 31, 'name': 'learn_entrant_syctomXV_111023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'photo_desc_type': 5972, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 3783, 'mtr_user_id': 31, 'name': 'learn_entrant_syctomXV_111023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'photo_desc_type': 5972, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5972 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5972, 'learn_entrant_syctomXV_111023', 16384, 25088, 'learn_entrant_syctomXV_111023', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 10, 11, 15, 57, 35), datetime.datetime(2023, 10, 11, 15, 57, 35)) {'thcl': {'id': 3783, 'mtr_user_id': 31, 'name': 'learn_entrant_syctomXV_111023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'photo_desc_type': 5972, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'sac', 'textiles', 'verre', 'organique', 'dasri', 'masque', 'encombrant', 'autre_emballage', 'autre_non_emballage', 'environnement'], 'list_hashtags_csv': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'svm_hashtag_type_desc': 5972, 'photo_desc_type': 5972, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'sac', 'textiles', 'verre', 'organique', 'dasri', 'masque', 'encombrant', 'autre_emballage', 'autre_non_emballage', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_entrant_syctomXV_111023 NUM_CLASSES 11 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_entrant_syctomXV_111023 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-03-10 20:01:49.904952: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-10 20:01:50.207365: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-10 20:01:51.845485: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:51.846341: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.60G (3865470464 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:52.756182: W tensorflow/core/common_runtime/bfc_allocator.cc:311] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature. 2025-03-10 20:01:52.835715: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:52.835867: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-10 20:01:52.837317: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:52.837348: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.29GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-10 20:01:52.845356: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:52.845458: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.78GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-10 20:01:52.846723: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:52.846746: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.78GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-10 20:01:52.891767: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:52.891869: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 63.85MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-10 20:01:52.893167: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:52.893198: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 63.85MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-10 20:01:52.894495: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:52.894523: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.26GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-10 20:01:52.895794: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:52.895826: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.26GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-10 20:01:52.905070: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:52.905130: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 417.62MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-10 20:01:52.906392: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:52.906419: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 417.62MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-03-10 20:01:52.924762: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:52.926258: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:52.927633: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:52.928998: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:52.934890: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:52.936386: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:52.937736: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:52.939067: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:52.941724: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:52.941846: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-03-10 20:01:52.951354: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:52.952247: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:52.961727: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:52.962586: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:52.963449: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:52.964382: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:52.965202: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-03-10 20:01:52.966000: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/learn_entrant_syctomXV_111023 /data/models_weight/learn_entrant_syctomXV_111023/mask_model.h5 size_local : 256052544 size in s3 : 256052544 create time local : 2023-10-26 16:36:43 create time in s3 : 2023-10-11 13:57:17 mask_model.h5 already exist and didn't need to update list_images length : 4 NEW PHOTO Processing 1 images image shape: (4160, 3120, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 150.72500 image_metas shape: (1, 19) min: 0.00000 max: 4160.00000 nb d'objets trouves : 7 NEW PHOTO Processing 1 images image shape: (4160, 3120, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 19) min: 0.00000 max: 4160.00000 nb d'objets trouves : 18 NEW PHOTO Processing 1 images image shape: (4160, 3120, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 19) min: 0.00000 max: 4160.00000 nb d'objets trouves : 19 NEW PHOTO Processing 1 images image shape: (4160, 3120, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 19) min: 0.00000 max: 4160.00000 nb d'objets trouves : 4 Detection mask done ! Trying to reset tf kernel 471892 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 3667 tf kernel not reseted sub process len(results) : 4 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 4 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 4860 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl3783 Catched exception ! Connect or reconnect ! thcls : [{'id': 3783, 'mtr_user_id': 31, 'name': 'learn_entrant_syctomXV_111023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'photo_desc_type': 5972, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 3783, 'mtr_user_id': 31, 'name': 'learn_entrant_syctomXV_111023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'photo_desc_type': 5972, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5972 ['background', 'sac', 'textiles', 'verre', 'organique', 'dasri', 'masque', 'encombrant', 'autre_emballage', 'autre_non_emballage', 'environnement'] time for calcul the mask position with numpy : 0.0009055137634277344 nb_pixel_total : 33324 time to create 1 rle with old method : 0.04260897636413574 length of segment : 251 time for calcul the mask position with numpy : 0.0006666183471679688 nb_pixel_total : 29217 time to create 1 rle with old method : 0.033998727798461914 length of segment : 190 time for calcul the mask position with numpy : 0.003384828567504883 nb_pixel_total : 62225 time to create 1 rle with old method : 0.07389497756958008 length of segment : 390 time for calcul the mask position with numpy : 0.000537872314453125 nb_pixel_total : 20404 time to create 1 rle with old method : 0.02487015724182129 length of segment : 111 time for calcul the mask position with numpy : 0.024646759033203125 nb_pixel_total : 471327 time to create 1 rle with new method : 0.04953408241271973 length of segment : 1085 time for calcul the mask position with numpy : 0.0005125999450683594 nb_pixel_total : 16493 time to create 1 rle with old method : 0.021021604537963867 length of segment : 224 time for calcul the mask position with numpy : 0.0007498264312744141 nb_pixel_total : 25497 time to create 1 rle with old method : 0.03015446662902832 length of segment : 436 time for calcul the mask position with numpy : 0.003601551055908203 nb_pixel_total : 75492 time to create 1 rle with old method : 0.11489367485046387 length of segment : 310 time for calcul the mask position with numpy : 0.0026345252990722656 nb_pixel_total : 81931 time to create 1 rle with old method : 0.1127469539642334 length of segment : 477 time for calcul the mask position with numpy : 0.0038716793060302734 nb_pixel_total : 176489 time to create 1 rle with new method : 0.0072171688079833984 length of segment : 565 time for calcul the mask position with numpy : 0.002564668655395508 nb_pixel_total : 74220 time to create 1 rle with old method : 0.08700418472290039 length of segment : 353 time for calcul the mask position with numpy : 0.0017900466918945312 nb_pixel_total : 61001 time to create 1 rle with old method : 0.07531619071960449 length of segment : 253 time for calcul the mask position with numpy : 0.0008461475372314453 nb_pixel_total : 31274 time to create 1 rle with old method : 0.03691458702087402 length of segment : 283 time for calcul the mask position with numpy : 0.0027348995208740234 nb_pixel_total : 70338 time to create 1 rle with old method : 0.08398056030273438 length of segment : 368 time for calcul the mask position with numpy : 0.002590656280517578 nb_pixel_total : 158354 time to create 1 rle with new method : 0.005706310272216797 length of segment : 467 time for calcul the mask position with numpy : 0.0030384063720703125 nb_pixel_total : 84398 time to create 1 rle with old method : 0.09651875495910645 length of segment : 541 time for calcul the mask position with numpy : 0.0013256072998046875 nb_pixel_total : 51206 time to create 1 rle with old method : 0.060009002685546875 length of segment : 305 time for calcul the mask position with numpy : 0.002592802047729492 nb_pixel_total : 123397 time to create 1 rle with old method : 0.14443445205688477 length of segment : 390 time for calcul the mask position with numpy : 0.0004088878631591797 nb_pixel_total : 23554 time to create 1 rle with old method : 0.027664899826049805 length of segment : 117 time for calcul the mask position with numpy : 0.0018770694732666016 nb_pixel_total : 85991 time to create 1 rle with old method : 0.10347127914428711 length of segment : 475 time for calcul the mask position with numpy : 0.0018727779388427734 nb_pixel_total : 63228 time to create 1 rle with old method : 0.07716917991638184 length of segment : 339 time for calcul the mask position with numpy : 0.002104520797729492 nb_pixel_total : 72760 time to create 1 rle with old method : 0.08513307571411133 length of segment : 239 time for calcul the mask position with numpy : 0.0014827251434326172 nb_pixel_total : 76493 time to create 1 rle with old method : 0.09092283248901367 length of segment : 662 time for calcul the mask position with numpy : 0.0021402835845947266 nb_pixel_total : 54203 time to create 1 rle with old method : 0.06513237953186035 length of segment : 306 time for calcul the mask position with numpy : 0.001195669174194336 nb_pixel_total : 33763 time to create 1 rle with old method : 0.040337324142456055 length of segment : 249 time for calcul the mask position with numpy : 0.0034019947052001953 nb_pixel_total : 123972 time to create 1 rle with old method : 0.14453530311584473 length of segment : 434 time for calcul the mask position with numpy : 0.0010101795196533203 nb_pixel_total : 28467 time to create 1 rle with old method : 0.03311634063720703 length of segment : 268 time for calcul the mask position with numpy : 0.001989126205444336 nb_pixel_total : 77440 time to create 1 rle with old method : 0.09387087821960449 length of segment : 253 time for calcul the mask position with numpy : 0.003020048141479492 nb_pixel_total : 118206 time to create 1 rle with old method : 0.13618135452270508 length of segment : 470 time for calcul the mask position with numpy : 0.0013234615325927734 nb_pixel_total : 38649 time to create 1 rle with old method : 0.04555773735046387 length of segment : 246 time for calcul the mask position with numpy : 0.0010769367218017578 nb_pixel_total : 31478 time to create 1 rle with old method : 0.037557125091552734 length of segment : 193 time for calcul the mask position with numpy : 0.0005347728729248047 nb_pixel_total : 8427 time to create 1 rle with old method : 0.010790824890136719 length of segment : 220 time for calcul the mask position with numpy : 0.004972219467163086 nb_pixel_total : 182325 time to create 1 rle with new method : 0.009896993637084961 length of segment : 481 time for calcul the mask position with numpy : 0.00026607513427734375 nb_pixel_total : 825 time to create 1 rle with old method : 0.001215219497680664 length of segment : 26 time for calcul the mask position with numpy : 0.001058340072631836 nb_pixel_total : 24046 time to create 1 rle with old method : 0.029488325119018555 length of segment : 173 time for calcul the mask position with numpy : 0.0031015872955322266 nb_pixel_total : 84807 time to create 1 rle with old method : 0.13932561874389648 length of segment : 352 time for calcul the mask position with numpy : 0.0022792816162109375 nb_pixel_total : 69151 time to create 1 rle with old method : 0.08261632919311523 length of segment : 278 time for calcul the mask position with numpy : 0.013748407363891602 nb_pixel_total : 627475 time to create 1 rle with new method : 0.06935954093933105 length of segment : 755 time for calcul the mask position with numpy : 0.0034627914428710938 nb_pixel_total : 160979 time to create 1 rle with new method : 0.006451606750488281 length of segment : 438 time for calcul the mask position with numpy : 0.0012617111206054688 nb_pixel_total : 32219 time to create 1 rle with old method : 0.03771829605102539 length of segment : 391 time for calcul the mask position with numpy : 0.013852357864379883 nb_pixel_total : 400412 time to create 1 rle with new method : 0.0853114128112793 length of segment : 1014 time for calcul the mask position with numpy : 0.001743316650390625 nb_pixel_total : 44848 time to create 1 rle with old method : 0.057425737380981445 length of segment : 281 time for calcul the mask position with numpy : 0.0012543201446533203 nb_pixel_total : 30012 time to create 1 rle with old method : 0.036107778549194336 length of segment : 305 time for calcul the mask position with numpy : 0.03211045265197754 nb_pixel_total : 675707 time to create 1 rle with new method : 0.40712547302246094 length of segment : 784 time for calcul the mask position with numpy : 0.7229995727539062 nb_pixel_total : 6508027 time to create 1 rle with new method : 1.0670623779296875 length of segment : 4744 time for calcul the mask position with numpy : 0.32022714614868164 nb_pixel_total : 3969420 time to create 1 rle with new method : 0.26537132263183594 length of segment : 1628 time for calcul the mask position with numpy : 0.21841859817504883 nb_pixel_total : 5820280 time to create 1 rle with new method : 0.37347960472106934 length of segment : 2225 time for calcul the mask position with numpy : 0.2235410213470459 nb_pixel_total : 5236542 time to create 1 rle with new method : 0.3406996726989746 length of segment : 1943 time spent for convertir_results : 14.389870882034302 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 48 chid ids of type : 4854 Number RLEs to save : 0 save missing photos in datou_result : time spend for datou_step_exec : 51.379329681396484 time spend to save output : 0.2656421661376953 total time spend for step 1 : 51.64497184753418 step2:mask_detect Mon Mar 10 20:02:23 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 4860 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-03-10 20:02:28.250688: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-03-10 20:02:28.283311: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-03-10 20:02:28.285792: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fe210000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-03-10 20:02:28.285856: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-03-10 20:02:28.290358: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-03-10 20:02:28.586255: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x11b51fd0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-03-10 20:02:28.586319: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-03-10 20:02:28.587277: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-10 20:02:28.588336: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-10 20:02:28.594271: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-10 20:02:28.603294: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-10 20:02:28.605017: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-10 20:02:28.614600: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-10 20:02:28.619299: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-10 20:02:28.638870: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-10 20:02:28.641018: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-10 20:02:28.641159: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-10 20:02:28.642213: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-10 20:02:28.642274: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-10 20:02:28.642285: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-10 20:02:28.649576: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9398 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-03-10 20:02:28.934873: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-10 20:02:28.935062: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-10 20:02:28.935090: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-10 20:02:28.935114: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-10 20:02:28.935137: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-10 20:02:28.935160: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-10 20:02:28.935184: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-10 20:02:28.935207: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-10 20:02:28.936893: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-10 20:02:28.938704: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-03-10 20:02:28.938830: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-03-10 20:02:28.938856: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-10 20:02:28.938880: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-03-10 20:02:28.938919: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-03-10 20:02:28.938947: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-03-10 20:02:28.938970: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-03-10 20:02:28.938993: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-03-10 20:02:28.940786: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-03-10 20:02:28.940872: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-03-10 20:02:28.940886: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-03-10 20:02:28.940897: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-03-10 20:02:28.942690: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9398 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl3453 thcls : [{'id': 3453, 'mtr_user_id': 31, 'name': 'mask_refus_amiens_050123', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4485, 'photo_desc_type': 5732, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}] thcl {'id': 3453, 'mtr_user_id': 31, 'name': 'mask_refus_amiens_050123', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4485, 'photo_desc_type': 5732, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5732 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5732, 'mask_refus_amiens_050123', 16384, 25088, 'mask_refus_amiens_050123', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 1, 5, 13, 23, 59), datetime.datetime(2023, 1, 5, 13, 23, 59)) {'thcl': {'id': 3453, 'mtr_user_id': 31, 'name': 'mask_refus_amiens_050123', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4485, 'photo_desc_type': 5732, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'], 'list_hashtags_csv': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4485, 'svm_hashtag_type_desc': 5732, 'photo_desc_type': 5732, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_refus_amiens_050123 NUM_CLASSES 9 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : mask_refus_amiens_050123 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-03-10 20:02:43.158805: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-03-10 20:02:43.370791: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/mask_refus_amiens_050123 /data/models_weight/mask_refus_amiens_050123/mask_model.h5 size_local : 256009536 size in s3 : 256009536 create time local : 2023-01-05 13:44:07 create time in s3 : 2023-01-05 12:23:52 mask_model.h5 already exist and didn't need to update list_images length : 4 NEW PHOTO Processing 1 images image shape: (4160, 3120, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 150.72500 image_metas shape: (1, 17) min: 0.00000 max: 4160.00000 nb d'objets trouves : 62 NEW PHOTO Processing 1 images image shape: (4160, 3120, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 4160.00000 nb d'objets trouves : 78 NEW PHOTO Processing 1 images image shape: (4160, 3120, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 4160.00000 nb d'objets trouves : 51 NEW PHOTO Processing 1 images image shape: (4160, 3120, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 4160.00000 nb d'objets trouves : 5 Detection mask done ! Trying to reset tf kernel 474386 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 4860 tf kernel not reseted sub process len(results) : 4 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 4 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 10149 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl3453 Catched exception ! Connect or reconnect ! thcls : [{'id': 3453, 'mtr_user_id': 31, 'name': 'mask_refus_amiens_050123', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4485, 'photo_desc_type': 5732, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}] thcl {'id': 3453, 'mtr_user_id': 31, 'name': 'mask_refus_amiens_050123', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4485, 'photo_desc_type': 5732, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5732 ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'] time for calcul the mask position with numpy : 0.005789756774902344 nb_pixel_total : 125453 time to create 1 rle with old method : 0.15648698806762695 length of segment : 664 time for calcul the mask position with numpy : 0.0020456314086914062 nb_pixel_total : 103324 time to create 1 rle with old method : 0.1509695053100586 length of segment : 272 time for calcul the mask position with numpy : 0.001332998275756836 nb_pixel_total : 34950 time to create 1 rle with old method : 0.04089546203613281 length of segment : 467 time for calcul the mask position with numpy : 0.008811712265014648 nb_pixel_total : 390455 time to create 1 rle with new method : 0.032224416732788086 length of segment : 566 time for calcul the mask position with numpy : 0.0008783340454101562 nb_pixel_total : 41330 time to create 1 rle with old method : 0.0490565299987793 length of segment : 291 time for calcul the mask position with numpy : 0.000652313232421875 nb_pixel_total : 31293 time to create 1 rle with old method : 0.03804183006286621 length of segment : 135 time for calcul the mask position with numpy : 0.0021064281463623047 nb_pixel_total : 87565 time to create 1 rle with old method : 0.10634517669677734 length of segment : 346 time for calcul the mask position with numpy : 0.003628253936767578 nb_pixel_total : 173704 time to create 1 rle with new method : 0.008470535278320312 length of segment : 448 time for calcul the mask position with numpy : 0.011893510818481445 nb_pixel_total : 420456 time to create 1 rle with new method : 0.05030488967895508 length of segment : 956 time for calcul the mask position with numpy : 0.008165121078491211 nb_pixel_total : 362171 time to create 1 rle with new method : 0.023043155670166016 length of segment : 744 time for calcul the mask position with numpy : 0.00771784782409668 nb_pixel_total : 268134 time to create 1 rle with new method : 0.020383119583129883 length of segment : 835 time for calcul the mask position with numpy : 0.004839181900024414 nb_pixel_total : 215452 time to create 1 rle with new method : 0.011658430099487305 length of segment : 664 time for calcul the mask position with numpy : 0.006608009338378906 nb_pixel_total : 286305 time to create 1 rle with new method : 0.017284154891967773 length of segment : 598 time for calcul the mask position with numpy : 0.00046253204345703125 nb_pixel_total : 13545 time to create 1 rle with old method : 0.016330480575561523 length of segment : 191 time for calcul the mask position with numpy : 0.0014154911041259766 nb_pixel_total : 35988 time to create 1 rle with old method : 0.04350590705871582 length of segment : 440 time for calcul the mask position with numpy : 0.0069010257720947266 nb_pixel_total : 291433 time to create 1 rle with new method : 0.019060850143432617 length of segment : 957 time for calcul the mask position with numpy : 0.0019812583923339844 nb_pixel_total : 75812 time to create 1 rle with old method : 0.08961653709411621 length of segment : 516 time for calcul the mask position with numpy : 0.001016378402709961 nb_pixel_total : 44955 time to create 1 rle with old method : 0.05308270454406738 length of segment : 329 time for calcul the mask position with numpy : 0.008923530578613281 nb_pixel_total : 384802 time to create 1 rle with new method : 0.026786327362060547 length of segment : 729 time for calcul the mask position with numpy : 0.005132198333740234 nb_pixel_total : 259557 time to create 1 rle with new method : 0.014239072799682617 length of segment : 1210 time for calcul the mask position with numpy : 0.0014557838439941406 nb_pixel_total : 62947 time to create 1 rle with old method : 0.09409499168395996 length of segment : 334 time for calcul the mask position with numpy : 0.0013189315795898438 nb_pixel_total : 41516 time to create 1 rle with old method : 0.05275225639343262 length of segment : 323 time for calcul the mask position with numpy : 0.000823974609375 nb_pixel_total : 27865 time to create 1 rle with old method : 0.03633236885070801 length of segment : 136 time for calcul the mask position with numpy : 0.0011839866638183594 nb_pixel_total : 44543 time to create 1 rle with old method : 0.055056095123291016 length of segment : 323 time for calcul the mask position with numpy : 0.0012962818145751953 nb_pixel_total : 52374 time to create 1 rle with old method : 0.061521291732788086 length of segment : 230 time for calcul the mask position with numpy : 0.002627849578857422 nb_pixel_total : 103754 time to create 1 rle with old method : 0.12099242210388184 length of segment : 598 time for calcul the mask position with numpy : 0.0011248588562011719 nb_pixel_total : 29133 time to create 1 rle with old method : 0.03998732566833496 length of segment : 311 time for calcul the mask position with numpy : 0.003957033157348633 nb_pixel_total : 162726 time to create 1 rle with new method : 0.011475324630737305 length of segment : 527 time for calcul the mask position with numpy : 0.001188516616821289 nb_pixel_total : 55607 time to create 1 rle with old method : 0.06519436836242676 length of segment : 371 time for calcul the mask position with numpy : 0.0011725425720214844 nb_pixel_total : 46522 time to create 1 rle with old method : 0.05449104309082031 length of segment : 321 time for calcul the mask position with numpy : 0.004554033279418945 nb_pixel_total : 195266 time to create 1 rle with new method : 0.013307571411132812 length of segment : 505 time for calcul the mask position with numpy : 0.0019855499267578125 nb_pixel_total : 62404 time to create 1 rle with old method : 0.07673311233520508 length of segment : 295 time for calcul the mask position with numpy : 0.0005562305450439453 nb_pixel_total : 18985 time to create 1 rle with old method : 0.022419452667236328 length of segment : 183 time for calcul the mask position with numpy : 0.0048863887786865234 nb_pixel_total : 225313 time to create 1 rle with new method : 0.013312339782714844 length of segment : 1022 time for calcul the mask position with numpy : 0.00237274169921875 nb_pixel_total : 103489 time to create 1 rle with old method : 0.11946868896484375 length of segment : 410 time for calcul the mask position with numpy : 0.0013108253479003906 nb_pixel_total : 53826 time to create 1 rle with old method : 0.06590795516967773 length of segment : 195 time for calcul the mask position with numpy : 0.010766983032226562 nb_pixel_total : 332027 time to create 1 rle with new method : 0.029053688049316406 length of segment : 816 time for calcul the mask position with numpy : 0.001373291015625 nb_pixel_total : 61704 time to create 1 rle with old method : 0.07131600379943848 length of segment : 269 time for calcul the mask position with numpy : 0.0018842220306396484 nb_pixel_total : 61582 time to create 1 rle with old method : 0.07575583457946777 length of segment : 352 time for calcul the mask position with numpy : 0.0014262199401855469 nb_pixel_total : 52628 time to create 1 rle with old method : 0.0744624137878418 length of segment : 364 time for calcul the mask position with numpy : 0.0006990432739257812 nb_pixel_total : 21837 time to create 1 rle with old method : 0.025975942611694336 length of segment : 247 time for calcul the mask position with numpy : 0.006342411041259766 nb_pixel_total : 293533 time to create 1 rle with new method : 0.016719341278076172 length of segment : 593 time for calcul the mask position with numpy : 0.011069774627685547 nb_pixel_total : 518854 time to create 1 rle with new method : 0.027535200119018555 length of segment : 1089 time for calcul the mask position with numpy : 0.0006010532379150391 nb_pixel_total : 13937 time to create 1 rle with old method : 0.01692032814025879 length of segment : 110 time for calcul the mask position with numpy : 0.0031960010528564453 nb_pixel_total : 133591 time to create 1 rle with old method : 0.15395164489746094 length of segment : 450 time for calcul the mask position with numpy : 0.0030286312103271484 nb_pixel_total : 137091 time to create 1 rle with old method : 0.16097784042358398 length of segment : 453 time for calcul the mask position with numpy : 0.002637147903442383 nb_pixel_total : 117014 time to create 1 rle with old method : 0.1363983154296875 length of segment : 414 time for calcul the mask position with numpy : 0.0016961097717285156 nb_pixel_total : 65036 time to create 1 rle with old method : 0.09647130966186523 length of segment : 300 time for calcul the mask position with numpy : 0.0040569305419921875 nb_pixel_total : 181130 time to create 1 rle with new method : 0.011212348937988281 length of segment : 648 time for calcul the mask position with numpy : 0.0006265640258789062 nb_pixel_total : 12648 time to create 1 rle with old method : 0.015143871307373047 length of segment : 147 time for calcul the mask position with numpy : 0.016248703002929688 nb_pixel_total : 462789 time to create 1 rle with new method : 0.07902646064758301 length of segment : 598 time for calcul the mask position with numpy : 0.0036177635192871094 nb_pixel_total : 180790 time to create 1 rle with new method : 0.009789705276489258 length of segment : 452 time for calcul the mask position with numpy : 0.0019316673278808594 nb_pixel_total : 86632 time to create 1 rle with old method : 0.10339736938476562 length of segment : 332 time for calcul the mask position with numpy : 0.0030553340911865234 nb_pixel_total : 86757 time to create 1 rle with old method : 0.10202813148498535 length of segment : 390 time for calcul the mask position with numpy : 0.0020241737365722656 nb_pixel_total : 85606 time to create 1 rle with old method : 0.0981593132019043 length of segment : 340 time for calcul the mask position with numpy : 0.004323720932006836 nb_pixel_total : 132443 time to create 1 rle with old method : 0.19923877716064453 length of segment : 620 time for calcul the mask position with numpy : 0.002603292465209961 nb_pixel_total : 113782 time to create 1 rle with old method : 0.12955498695373535 length of segment : 269 time for calcul the mask position with numpy : 0.0021209716796875 nb_pixel_total : 91944 time to create 1 rle with old method : 0.10607290267944336 length of segment : 364 time for calcul the mask position with numpy : 0.001737833023071289 nb_pixel_total : 54762 time to create 1 rle with old method : 0.06485104560852051 length of segment : 234 time for calcul the mask position with numpy : 0.001760244369506836 nb_pixel_total : 74541 time to create 1 rle with old method : 0.08570718765258789 length of segment : 347 time for calcul the mask position with numpy : 0.005646944046020508 nb_pixel_total : 305800 time to create 1 rle with new method : 0.01631760597229004 length of segment : 1168 time for calcul the mask position with numpy : 0.002276182174682617 nb_pixel_total : 114481 time to create 1 rle with old method : 0.1306769847869873 length of segment : 282 time for calcul the mask position with numpy : 0.0010647773742675781 nb_pixel_total : 40624 time to create 1 rle with old method : 0.04706120491027832 length of segment : 240 time for calcul the mask position with numpy : 0.003810405731201172 nb_pixel_total : 187692 time to create 1 rle with new method : 0.010698080062866211 length of segment : 668 time for calcul the mask position with numpy : 0.018705368041992188 nb_pixel_total : 633669 time to create 1 rle with new method : 0.04896664619445801 length of segment : 814 time for calcul the mask position with numpy : 0.002198457717895508 nb_pixel_total : 66562 time to create 1 rle with old method : 0.07666254043579102 length of segment : 422 time for calcul the mask position with numpy : 0.002834796905517578 nb_pixel_total : 142830 time to create 1 rle with old method : 0.1665961742401123 length of segment : 456 time for calcul the mask position with numpy : 0.004405975341796875 nb_pixel_total : 158937 time to create 1 rle with new method : 0.012465715408325195 length of segment : 737 time for calcul the mask position with numpy : 0.0007970333099365234 nb_pixel_total : 34305 time to create 1 rle with old method : 0.05539822578430176 length of segment : 212 time for calcul the mask position with numpy : 0.0035343170166015625 nb_pixel_total : 115152 time to create 1 rle with old method : 0.13126015663146973 length of segment : 433 time for calcul the mask position with numpy : 0.002518892288208008 nb_pixel_total : 76427 time to create 1 rle with old method : 0.0890188217163086 length of segment : 579 time for calcul the mask position with numpy : 0.003909587860107422 nb_pixel_total : 188063 time to create 1 rle with new method : 0.011506795883178711 length of segment : 670 time for calcul the mask position with numpy : 0.0009768009185791016 nb_pixel_total : 48446 time to create 1 rle with old method : 0.0595245361328125 length of segment : 320 time for calcul the mask position with numpy : 0.0050661563873291016 nb_pixel_total : 215493 time to create 1 rle with new method : 0.014023542404174805 length of segment : 779 time for calcul the mask position with numpy : 0.001074075698852539 nb_pixel_total : 43104 time to create 1 rle with old method : 0.04923748970031738 length of segment : 312 time for calcul the mask position with numpy : 0.0025594234466552734 nb_pixel_total : 75078 time to create 1 rle with old method : 0.08537960052490234 length of segment : 493 time for calcul the mask position with numpy : 0.004259586334228516 nb_pixel_total : 117022 time to create 1 rle with old method : 0.1355304718017578 length of segment : 485 time for calcul the mask position with numpy : 0.003938913345336914 nb_pixel_total : 189809 time to create 1 rle with new method : 0.011146783828735352 length of segment : 544 time for calcul the mask position with numpy : 0.0011451244354248047 nb_pixel_total : 27931 time to create 1 rle with old method : 0.03259468078613281 length of segment : 288 time for calcul the mask position with numpy : 0.003070831298828125 nb_pixel_total : 88468 time to create 1 rle with old method : 0.10715222358703613 length of segment : 331 time for calcul the mask position with numpy : 0.001916646957397461 nb_pixel_total : 80586 time to create 1 rle with old method : 0.0939168930053711 length of segment : 294 time for calcul the mask position with numpy : 0.004261970520019531 nb_pixel_total : 220938 time to create 1 rle with new method : 0.011558771133422852 length of segment : 477 time for calcul the mask position with numpy : 0.0019497871398925781 nb_pixel_total : 70684 time to create 1 rle with old method : 0.0809938907623291 length of segment : 527 time for calcul the mask position with numpy : 0.0011334419250488281 nb_pixel_total : 30396 time to create 1 rle with old method : 0.03732872009277344 length of segment : 149 time for calcul the mask position with numpy : 0.0016663074493408203 nb_pixel_total : 80118 time to create 1 rle with old method : 0.09435248374938965 length of segment : 224 time for calcul the mask position with numpy : 0.0016696453094482422 nb_pixel_total : 57465 time to create 1 rle with old method : 0.06846928596496582 length of segment : 381 time for calcul the mask position with numpy : 0.009768486022949219 nb_pixel_total : 343558 time to create 1 rle with new method : 0.026923656463623047 length of segment : 1018 time for calcul the mask position with numpy : 0.00201416015625 nb_pixel_total : 74525 time to create 1 rle with old method : 0.08642721176147461 length of segment : 592 time for calcul the mask position with numpy : 0.0017189979553222656 nb_pixel_total : 30854 time to create 1 rle with old method : 0.036821603775024414 length of segment : 246 time for calcul the mask position with numpy : 0.0013082027435302734 nb_pixel_total : 31592 time to create 1 rle with old method : 0.037783145904541016 length of segment : 215 time for calcul the mask position with numpy : 0.0008718967437744141 nb_pixel_total : 42319 time to create 1 rle with old method : 0.049576759338378906 length of segment : 300 time for calcul the mask position with numpy : 0.0045261383056640625 nb_pixel_total : 215040 time to create 1 rle with new method : 0.011005401611328125 length of segment : 593 time for calcul the mask position with numpy : 0.0019276142120361328 nb_pixel_total : 67100 time to create 1 rle with old method : 0.0800471305847168 length of segment : 197 time for calcul the mask position with numpy : 0.0023326873779296875 nb_pixel_total : 72431 time to create 1 rle with old method : 0.08550882339477539 length of segment : 292 time for calcul the mask position with numpy : 0.0009891986846923828 nb_pixel_total : 46923 time to create 1 rle with old method : 0.0545346736907959 length of segment : 298 time for calcul the mask position with numpy : 0.002047300338745117 nb_pixel_total : 96713 time to create 1 rle with old method : 0.10937666893005371 length of segment : 424 time for calcul the mask position with numpy : 0.005224466323852539 nb_pixel_total : 250911 time to create 1 rle with new method : 0.01437234878540039 length of segment : 530 time for calcul the mask position with numpy : 0.0004031658172607422 nb_pixel_total : 12079 time to create 1 rle with old method : 0.014361143112182617 length of segment : 139 time for calcul the mask position with numpy : 0.0007519721984863281 nb_pixel_total : 17795 time to create 1 rle with old method : 0.020938396453857422 length of segment : 152 time for calcul the mask position with numpy : 0.0054471492767333984 nb_pixel_total : 259554 time to create 1 rle with new method : 0.014021158218383789 length of segment : 486 time for calcul the mask position with numpy : 0.00318145751953125 nb_pixel_total : 82293 time to create 1 rle with old method : 0.0937340259552002 length of segment : 465 time for calcul the mask position with numpy : 0.003013134002685547 nb_pixel_total : 131460 time to create 1 rle with old method : 0.1545872688293457 length of segment : 355 time for calcul the mask position with numpy : 0.007331371307373047 nb_pixel_total : 305415 time to create 1 rle with new method : 0.021317005157470703 length of segment : 1180 time for calcul the mask position with numpy : 0.0015807151794433594 nb_pixel_total : 50972 time to create 1 rle with old method : 0.05916166305541992 length of segment : 348 time for calcul the mask position with numpy : 0.003306865692138672 nb_pixel_total : 136277 time to create 1 rle with old method : 0.15923476219177246 length of segment : 352 time for calcul the mask position with numpy : 0.004357337951660156 nb_pixel_total : 120135 time to create 1 rle with old method : 0.13965606689453125 length of segment : 623 time for calcul the mask position with numpy : 0.0013747215270996094 nb_pixel_total : 42763 time to create 1 rle with old method : 0.048522233963012695 length of segment : 433 time for calcul the mask position with numpy : 0.012573719024658203 nb_pixel_total : 570584 time to create 1 rle with new method : 0.030016660690307617 length of segment : 1010 time for calcul the mask position with numpy : 0.0012362003326416016 nb_pixel_total : 48037 time to create 1 rle with old method : 0.07326531410217285 length of segment : 242 time for calcul the mask position with numpy : 0.0004138946533203125 nb_pixel_total : 11286 time to create 1 rle with old method : 0.013620615005493164 length of segment : 146 time for calcul the mask position with numpy : 0.004583120346069336 nb_pixel_total : 135767 time to create 1 rle with old method : 0.17072653770446777 length of segment : 600 time for calcul the mask position with numpy : 0.0010590553283691406 nb_pixel_total : 39431 time to create 1 rle with old method : 0.04635930061340332 length of segment : 234 time for calcul the mask position with numpy : 0.0021758079528808594 nb_pixel_total : 67330 time to create 1 rle with old method : 0.0792696475982666 length of segment : 283 time for calcul the mask position with numpy : 0.001851797103881836 nb_pixel_total : 75031 time to create 1 rle with old method : 0.08897781372070312 length of segment : 138 time for calcul the mask position with numpy : 0.002163410186767578 nb_pixel_total : 80795 time to create 1 rle with old method : 0.09779500961303711 length of segment : 458 time for calcul the mask position with numpy : 0.02433180809020996 nb_pixel_total : 416109 time to create 1 rle with new method : 0.15904951095581055 length of segment : 836 time for calcul the mask position with numpy : 0.006639003753662109 nb_pixel_total : 268992 time to create 1 rle with new method : 0.018586158752441406 length of segment : 710 time for calcul the mask position with numpy : 0.0011472702026367188 nb_pixel_total : 31264 time to create 1 rle with old method : 0.03721427917480469 length of segment : 235 time for calcul the mask position with numpy : 0.0068509578704833984 nb_pixel_total : 222410 time to create 1 rle with new method : 0.022286415100097656 length of segment : 929 time for calcul the mask position with numpy : 0.006913423538208008 nb_pixel_total : 351741 time to create 1 rle with new method : 0.022140979766845703 length of segment : 609 time for calcul the mask position with numpy : 0.0021982192993164062 nb_pixel_total : 94488 time to create 1 rle with old method : 0.11354565620422363 length of segment : 302 time for calcul the mask position with numpy : 0.0006668567657470703 nb_pixel_total : 20938 time to create 1 rle with old method : 0.024958372116088867 length of segment : 177 time for calcul the mask position with numpy : 0.030754566192626953 nb_pixel_total : 934996 time to create 1 rle with new method : 0.4855325222015381 length of segment : 1747 time for calcul the mask position with numpy : 0.003138303756713867 nb_pixel_total : 125482 time to create 1 rle with old method : 0.14377427101135254 length of segment : 495 time for calcul the mask position with numpy : 0.0017676353454589844 nb_pixel_total : 77400 time to create 1 rle with old method : 0.09574103355407715 length of segment : 257 time for calcul the mask position with numpy : 0.007088899612426758 nb_pixel_total : 183859 time to create 1 rle with new method : 0.024132251739501953 length of segment : 976 time for calcul the mask position with numpy : 0.002178192138671875 nb_pixel_total : 53046 time to create 1 rle with old method : 0.06802821159362793 length of segment : 457 time for calcul the mask position with numpy : 0.0021333694458007812 nb_pixel_total : 85641 time to create 1 rle with old method : 0.10298562049865723 length of segment : 505 time for calcul the mask position with numpy : 0.00928640365600586 nb_pixel_total : 244889 time to create 1 rle with new method : 0.025357961654663086 length of segment : 891 time for calcul the mask position with numpy : 0.0028357505798339844 nb_pixel_total : 73811 time to create 1 rle with old method : 0.08985233306884766 length of segment : 572 time for calcul the mask position with numpy : 0.0018651485443115234 nb_pixel_total : 43407 time to create 1 rle with old method : 0.05136990547180176 length of segment : 262 time for calcul the mask position with numpy : 0.0009107589721679688 nb_pixel_total : 30917 time to create 1 rle with old method : 0.03493452072143555 length of segment : 232 time for calcul the mask position with numpy : 0.018471717834472656 nb_pixel_total : 581743 time to create 1 rle with new method : 0.17943716049194336 length of segment : 948 time for calcul the mask position with numpy : 0.0030012130737304688 nb_pixel_total : 59392 time to create 1 rle with old method : 0.0668935775756836 length of segment : 342 time for calcul the mask position with numpy : 0.0057561397552490234 nb_pixel_total : 47777 time to create 1 rle with old method : 0.05569720268249512 length of segment : 227 time for calcul the mask position with numpy : 0.0017375946044921875 nb_pixel_total : 62706 time to create 1 rle with old method : 0.07418584823608398 length of segment : 558 time for calcul the mask position with numpy : 0.002234935760498047 nb_pixel_total : 53752 time to create 1 rle with old method : 0.06274986267089844 length of segment : 314 time for calcul the mask position with numpy : 0.0024650096893310547 nb_pixel_total : 72123 time to create 1 rle with old method : 0.08731627464294434 length of segment : 124 time for calcul the mask position with numpy : 0.008038759231567383 nb_pixel_total : 274844 time to create 1 rle with new method : 0.026724576950073242 length of segment : 669 time for calcul the mask position with numpy : 0.019790172576904297 nb_pixel_total : 642662 time to create 1 rle with new method : 0.06133222579956055 length of segment : 833 time for calcul the mask position with numpy : 0.0077669620513916016 nb_pixel_total : 376101 time to create 1 rle with new method : 0.018043041229248047 length of segment : 622 time for calcul the mask position with numpy : 0.01090240478515625 nb_pixel_total : 326571 time to create 1 rle with new method : 0.021490097045898438 length of segment : 556 time for calcul the mask position with numpy : 0.05783653259277344 nb_pixel_total : 826325 time to create 1 rle with new method : 0.21576809883117676 length of segment : 2275 time for calcul the mask position with numpy : 0.029835224151611328 nb_pixel_total : 773392 time to create 1 rle with new method : 0.05292868614196777 length of segment : 815 time for calcul the mask position with numpy : 0.0011119842529296875 nb_pixel_total : 25138 time to create 1 rle with old method : 0.03299260139465332 length of segment : 263 time for calcul the mask position with numpy : 0.005463600158691406 nb_pixel_total : 149135 time to create 1 rle with old method : 0.2080669403076172 length of segment : 645 time for calcul the mask position with numpy : 0.005616903305053711 nb_pixel_total : 196725 time to create 1 rle with new method : 0.013620138168334961 length of segment : 576 time for calcul the mask position with numpy : 0.008185386657714844 nb_pixel_total : 195080 time to create 1 rle with new method : 0.0207672119140625 length of segment : 488 time for calcul the mask position with numpy : 0.0008873939514160156 nb_pixel_total : 27700 time to create 1 rle with old method : 0.05127453804016113 length of segment : 204 time for calcul the mask position with numpy : 0.014274120330810547 nb_pixel_total : 222080 time to create 1 rle with new method : 0.02010488510131836 length of segment : 472 time for calcul the mask position with numpy : 0.005752086639404297 nb_pixel_total : 172886 time to create 1 rle with new method : 0.01703619956970215 length of segment : 699 time for calcul the mask position with numpy : 0.004278659820556641 nb_pixel_total : 93132 time to create 1 rle with old method : 0.12165522575378418 length of segment : 508 time for calcul the mask position with numpy : 0.0030727386474609375 nb_pixel_total : 122618 time to create 1 rle with old method : 0.1656491756439209 length of segment : 364 time for calcul the mask position with numpy : 0.008388280868530273 nb_pixel_total : 149051 time to create 1 rle with old method : 0.1774451732635498 length of segment : 493 time for calcul the mask position with numpy : 0.011755228042602539 nb_pixel_total : 188870 time to create 1 rle with new method : 0.013653278350830078 length of segment : 656 time for calcul the mask position with numpy : 0.00543665885925293 nb_pixel_total : 137166 time to create 1 rle with old method : 0.16689586639404297 length of segment : 343 time for calcul the mask position with numpy : 0.00700831413269043 nb_pixel_total : 155286 time to create 1 rle with new method : 0.009508371353149414 length of segment : 397 time for calcul the mask position with numpy : 0.002270221710205078 nb_pixel_total : 116395 time to create 1 rle with old method : 0.1356046199798584 length of segment : 642 time for calcul the mask position with numpy : 0.0006997585296630859 nb_pixel_total : 17029 time to create 1 rle with old method : 0.02927541732788086 length of segment : 158 time for calcul the mask position with numpy : 0.0028333663940429688 nb_pixel_total : 41126 time to create 1 rle with old method : 0.049054622650146484 length of segment : 219 time for calcul the mask position with numpy : 0.0030846595764160156 nb_pixel_total : 124944 time to create 1 rle with old method : 0.14393997192382812 length of segment : 489 time for calcul the mask position with numpy : 0.0017578601837158203 nb_pixel_total : 68726 time to create 1 rle with old method : 0.10417675971984863 length of segment : 339 time for calcul the mask position with numpy : 0.002201557159423828 nb_pixel_total : 56874 time to create 1 rle with old method : 0.07320141792297363 length of segment : 254 time for calcul the mask position with numpy : 0.0015501976013183594 nb_pixel_total : 62435 time to create 1 rle with old method : 0.08016061782836914 length of segment : 330 time for calcul the mask position with numpy : 0.00867772102355957 nb_pixel_total : 156686 time to create 1 rle with new method : 0.012154817581176758 length of segment : 453 time for calcul the mask position with numpy : 0.029735088348388672 nb_pixel_total : 575193 time to create 1 rle with new method : 0.06041288375854492 length of segment : 716 time for calcul the mask position with numpy : 0.0339505672454834 nb_pixel_total : 818813 time to create 1 rle with new method : 0.4209020137786865 length of segment : 1219 time for calcul the mask position with numpy : 0.0007050037384033203 nb_pixel_total : 16772 time to create 1 rle with old method : 0.020587921142578125 length of segment : 127 time for calcul the mask position with numpy : 0.0008766651153564453 nb_pixel_total : 45386 time to create 1 rle with old method : 0.05668044090270996 length of segment : 241 time for calcul the mask position with numpy : 0.0005061626434326172 nb_pixel_total : 20883 time to create 1 rle with old method : 0.025732755661010742 length of segment : 254 time for calcul the mask position with numpy : 0.008517980575561523 nb_pixel_total : 479867 time to create 1 rle with new method : 0.024034500122070312 length of segment : 746 time for calcul the mask position with numpy : 0.0009872913360595703 nb_pixel_total : 39463 time to create 1 rle with old method : 0.050169944763183594 length of segment : 290 time for calcul the mask position with numpy : 0.0005791187286376953 nb_pixel_total : 28181 time to create 1 rle with old method : 0.033409833908081055 length of segment : 167 time for calcul the mask position with numpy : 0.00046706199645996094 nb_pixel_total : 18111 time to create 1 rle with old method : 0.022266149520874023 length of segment : 187 time for calcul the mask position with numpy : 0.003041982650756836 nb_pixel_total : 133318 time to create 1 rle with old method : 0.16225886344909668 length of segment : 404 time for calcul the mask position with numpy : 0.0017695426940917969 nb_pixel_total : 95943 time to create 1 rle with old method : 0.12616372108459473 length of segment : 347 time for calcul the mask position with numpy : 0.008714675903320312 nb_pixel_total : 274842 time to create 1 rle with new method : 0.04467129707336426 length of segment : 1241 time for calcul the mask position with numpy : 0.001135110855102539 nb_pixel_total : 30012 time to create 1 rle with old method : 0.035535573959350586 length of segment : 123 time for calcul the mask position with numpy : 0.0005867481231689453 nb_pixel_total : 22553 time to create 1 rle with old method : 0.02642989158630371 length of segment : 196 time for calcul the mask position with numpy : 0.0018508434295654297 nb_pixel_total : 38689 time to create 1 rle with old method : 0.0450444221496582 length of segment : 161 time for calcul the mask position with numpy : 0.0023767948150634766 nb_pixel_total : 109872 time to create 1 rle with old method : 0.12660551071166992 length of segment : 403 time for calcul the mask position with numpy : 0.0007393360137939453 nb_pixel_total : 31816 time to create 1 rle with old method : 0.049010276794433594 length of segment : 160 time for calcul the mask position with numpy : 0.023392438888549805 nb_pixel_total : 588344 time to create 1 rle with new method : 0.09143567085266113 length of segment : 1577 time for calcul the mask position with numpy : 0.0019221305847167969 nb_pixel_total : 91149 time to create 1 rle with old method : 0.10801577568054199 length of segment : 266 time for calcul the mask position with numpy : 0.002323627471923828 nb_pixel_total : 115214 time to create 1 rle with old method : 0.13404250144958496 length of segment : 334 time for calcul the mask position with numpy : 0.011737585067749023 nb_pixel_total : 540808 time to create 1 rle with new method : 0.45705223083496094 length of segment : 706 time for calcul the mask position with numpy : 0.0019936561584472656 nb_pixel_total : 73658 time to create 1 rle with old method : 0.08554887771606445 length of segment : 288 time for calcul the mask position with numpy : 0.017062902450561523 nb_pixel_total : 236055 time to create 1 rle with new method : 0.1923823356628418 length of segment : 1320 time for calcul the mask position with numpy : 0.00385284423828125 nb_pixel_total : 176056 time to create 1 rle with new method : 0.010811328887939453 length of segment : 446 time for calcul the mask position with numpy : 0.003934621810913086 nb_pixel_total : 185774 time to create 1 rle with new method : 0.009864568710327148 length of segment : 463 time for calcul the mask position with numpy : 0.0008890628814697266 nb_pixel_total : 38404 time to create 1 rle with old method : 0.05218625068664551 length of segment : 256 time for calcul the mask position with numpy : 0.677527666091919 nb_pixel_total : 7392783 time to create 1 rle with new method : 1.2808258533477783 length of segment : 4102 time for calcul the mask position with numpy : 0.025830984115600586 nb_pixel_total : 1596500 time to create 1 rle with new method : 0.4561340808868408 length of segment : 1460 time for calcul the mask position with numpy : 0.0007009506225585938 nb_pixel_total : 52505 time to create 1 rle with old method : 0.061972856521606445 length of segment : 184 time for calcul the mask position with numpy : 0.020879745483398438 nb_pixel_total : 1150460 time to create 1 rle with new method : 0.2599484920501709 length of segment : 1917 time for calcul the mask position with numpy : 0.002905607223510742 nb_pixel_total : 155186 time to create 1 rle with new method : 0.005781650543212891 length of segment : 363 time spent for convertir_results : 41.67641282081604 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 244 chid ids of type : 4854 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Number RLEs to save : 0 save missing photos in datou_result : time spend for datou_step_exec : 123.43140840530396 time spend to save output : 0.5046520233154297 total time spend for step 2 : 123.93606042861938 step3:blur_detection Mon Mar 10 20:04:27 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure inside step blur_detection methode: variance laplacian (4160, 3120) 2014.5916170730059 (4160, 3120) 2073.371732719602 (4160, 3120) 3981.40409470297 (4160, 3120) 448.5099047918387 Inside saveOutput : final : False verbose : 0 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 4 time used for this insertion : 0.03236651420593262 begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 4 time used for this insertion : 0.03453469276428223 save missing photos in datou_result : time spend for datou_step_exec : 2.0945932865142822 time spend to save output : 0.0819101333618164 total time spend for step 3 : 2.1765034198760986 step4:brightness Mon Mar 10 20:04:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure inside step calcul brightness treat image : temp/1741633288_471565_1342523624_1d53e97de396f6d1cc06f02af26e7b79.jpg treat image : temp/1741633288_471565_1342523598_a0a92f3aaf05c23e365fcb4de4d08049.jpg treat image : temp/1741633288_471565_1342523593_6e0cda92f044c62cfa5ccfeb408823c2.jpg treat image : temp/1741633288_471565_1342523589_d6ea44040b83c00ec9aead9123e7958b.jpg Inside saveOutput : final : False verbose : 0 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 4 time used for this insertion : 0.045064449310302734 begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 4 time used for this insertion : 0.04351663589477539 save missing photos in datou_result : time spend for datou_step_exec : 6.89307975769043 time spend to save output : 0.1083371639251709 total time spend for step 4 : 7.001416921615601 step5:rle_unique_nms_with_priority Mon Mar 10 20:04:37 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms batch 1 Loaded 244 chid ids of type : 4854 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++nb_obj : 68 nb_hashtags : 8 time to prepare the origin masks : 55.38774228096008 time for calcul the mask position with numpy : 0.612804651260376 nb_pixel_total : 5040042 time to create 1 rle with new method : 2.0574350357055664 time for calcul the mask position with numpy : 0.05453157424926758 nb_pixel_total : 61670 time to create 1 rle with old method : 0.0687565803527832 time for calcul the mask position with numpy : 0.05624723434448242 nb_pixel_total : 109322 time to create 1 rle with old method : 0.14382481575012207 time for calcul the mask position with numpy : 0.05867934226989746 nb_pixel_total : 18935 time to create 1 rle with old method : 0.03188276290893555 time for calcul the mask position with numpy : 0.05764126777648926 nb_pixel_total : 44407 time to create 1 rle with old method : 0.052546024322509766 time for calcul the mask position with numpy : 0.05919241905212402 nb_pixel_total : 233888 time to create 1 rle with new method : 1.4768004417419434 time for calcul the mask position with numpy : 0.06206798553466797 nb_pixel_total : 31175 time to create 1 rle with old method : 0.036562442779541016 time for calcul the mask position with numpy : 0.1261882781982422 nb_pixel_total : 410211 time to create 1 rle with new method : 1.3626267910003662 time for calcul the mask position with numpy : 0.05396533012390137 nb_pixel_total : 7636 time to create 1 rle with old method : 0.009601116180419922 time for calcul the mask position with numpy : 0.055120229721069336 nb_pixel_total : 41431 time to create 1 rle with old method : 0.057564735412597656 time for calcul the mask position with numpy : 0.053655385971069336 nb_pixel_total : 102102 time to create 1 rle with old method : 0.14315462112426758 time for calcul the mask position with numpy : 0.05635690689086914 nb_pixel_total : 103217 time to create 1 rle with old method : 0.1251823902130127 time for calcul the mask position with numpy : 0.054998159408569336 nb_pixel_total : 62177 time to create 1 rle with old method : 0.07019281387329102 time for calcul the mask position with numpy : 0.05283522605895996 nb_pixel_total : 917 time to create 1 rle with old method : 0.0012288093566894531 time for calcul the mask position with numpy : 0.05363750457763672 nb_pixel_total : 27749 time to create 1 rle with old method : 0.03229236602783203 time for calcul the mask position with numpy : 0.05371665954589844 nb_pixel_total : 29089 time to create 1 rle with old method : 0.03281974792480469 time for calcul the mask position with numpy : 0.05330801010131836 nb_pixel_total : 44900 time to create 1 rle with old method : 0.05121350288391113 time for calcul the mask position with numpy : 0.05503225326538086 nb_pixel_total : 224781 time to create 1 rle with new method : 1.3922183513641357 time for calcul the mask position with numpy : 0.05632638931274414 nb_pixel_total : 384287 time to create 1 rle with new method : 1.1339542865753174 time for calcul the mask position with numpy : 0.053229331970214844 nb_pixel_total : 793 time to create 1 rle with old method : 0.0012013912200927734 time for calcul the mask position with numpy : 0.057350873947143555 nb_pixel_total : 83749 time to create 1 rle with old method : 0.09635257720947266 time for calcul the mask position with numpy : 0.05591392517089844 nb_pixel_total : 332418 time to create 1 rle with new method : 1.3980674743652344 time for calcul the mask position with numpy : 0.05357789993286133 nb_pixel_total : 13478 time to create 1 rle with old method : 0.016048431396484375 time for calcul the mask position with numpy : 0.05364561080932617 nb_pixel_total : 29128 time to create 1 rle with old method : 0.03474712371826172 time for calcul the mask position with numpy : 0.06024885177612305 nb_pixel_total : 18957 time to create 1 rle with old method : 0.023334503173828125 time for calcul the mask position with numpy : 0.05281186103820801 nb_pixel_total : 33222 time to create 1 rle with old method : 0.03972172737121582 time for calcul the mask position with numpy : 0.0585024356842041 nb_pixel_total : 75467 time to create 1 rle with old method : 0.08729362487792969 time for calcul the mask position with numpy : 0.05262565612792969 nb_pixel_total : 173385 time to create 1 rle with new method : 1.3937902450561523 time for calcul the mask position with numpy : 0.054315805435180664 nb_pixel_total : 13817 time to create 1 rle with old method : 0.016206741333007812 time for calcul the mask position with numpy : 0.05785179138183594 nb_pixel_total : 102958 time to create 1 rle with old method : 0.11660218238830566 time for calcul the mask position with numpy : 0.05395340919494629 nb_pixel_total : 61618 time to create 1 rle with old method : 0.07820272445678711 time for calcul the mask position with numpy : 0.05305361747741699 nb_pixel_total : 22075 time to create 1 rle with old method : 0.026070117950439453 time for calcul the mask position with numpy : 0.05345916748046875 nb_pixel_total : 193224 time to create 1 rle with new method : 1.5043160915374756 time for calcul the mask position with numpy : 0.05555558204650879 nb_pixel_total : 284955 time to create 1 rle with new method : 1.6839840412139893 time for calcul the mask position with numpy : 0.05408334732055664 nb_pixel_total : 55505 time to create 1 rle with old method : 0.06978034973144531 time for calcul the mask position with numpy : 0.05436348915100098 nb_pixel_total : 5539 time to create 1 rle with old method : 0.006880283355712891 time for calcul the mask position with numpy : 0.05399918556213379 nb_pixel_total : 133576 time to create 1 rle with old method : 0.1512455940246582 time for calcul the mask position with numpy : 0.054257869720458984 nb_pixel_total : 87432 time to create 1 rle with old method : 0.10841774940490723 time for calcul the mask position with numpy : 0.06911563873291016 nb_pixel_total : 361869 time to create 1 rle with new method : 2.0814192295074463 time for calcul the mask position with numpy : 0.05505108833312988 nb_pixel_total : 179387 time to create 1 rle with new method : 1.8673732280731201 time for calcul the mask position with numpy : 0.06274032592773438 nb_pixel_total : 420129 time to create 1 rle with new method : 2.222872495651245 time for calcul the mask position with numpy : 0.05433011054992676 nb_pixel_total : 46311 time to create 1 rle with old method : 0.06278347969055176 time for calcul the mask position with numpy : 0.05409407615661621 nb_pixel_total : 62877 time to create 1 rle with old method : 0.07639956474304199 time for calcul the mask position with numpy : 0.057450294494628906 nb_pixel_total : 115665 time to create 1 rle with old method : 0.15665221214294434 time for calcul the mask position with numpy : 0.054656982421875 nb_pixel_total : 84971 time to create 1 rle with old method : 0.09737277030944824 time for calcul the mask position with numpy : 0.054657936096191406 nb_pixel_total : 41653 time to create 1 rle with old method : 0.0679781436920166 time for calcul the mask position with numpy : 0.062142133712768555 nb_pixel_total : 389783 time to create 1 rle with new method : 1.1492981910705566 time for calcul the mask position with numpy : 0.05817461013793945 nb_pixel_total : 144763 time to create 1 rle with old method : 0.17511653900146484 time for calcul the mask position with numpy : 0.05477738380432129 nb_pixel_total : 12579 time to create 1 rle with old method : 0.014589071273803711 time for calcul the mask position with numpy : 0.053600311279296875 nb_pixel_total : 14707 time to create 1 rle with old method : 0.017285823822021484 time for calcul the mask position with numpy : 0.05475044250488281 nb_pixel_total : 162376 time to create 1 rle with new method : 1.5156266689300537 time for calcul the mask position with numpy : 0.05593109130859375 nb_pixel_total : 259549 time to create 1 rle with new method : 2.2882485389709473 time for calcul the mask position with numpy : 0.05424642562866211 nb_pixel_total : 59885 time to create 1 rle with old method : 0.0712592601776123 time for calcul the mask position with numpy : 0.05379009246826172 nb_pixel_total : 3756 time to create 1 rle with old method : 0.004937171936035156 time for calcul the mask position with numpy : 0.05346941947937012 nb_pixel_total : 13967 time to create 1 rle with old method : 0.0174560546875 time for calcul the mask position with numpy : 0.05960536003112793 nb_pixel_total : 511207 time to create 1 rle with new method : 1.6490695476531982 time for calcul the mask position with numpy : 0.054624319076538086 nb_pixel_total : 153737 time to create 1 rle with new method : 1.4583559036254883 time for calcul the mask position with numpy : 0.05668520927429199 nb_pixel_total : 54160 time to create 1 rle with old method : 0.06824302673339844 time for calcul the mask position with numpy : 0.05537152290344238 nb_pixel_total : 16347 time to create 1 rle with old method : 0.020240068435668945 time for calcul the mask position with numpy : 0.057947397232055664 nb_pixel_total : 214827 time to create 1 rle with new method : 1.3813869953155518 time for calcul the mask position with numpy : 0.055872440338134766 nb_pixel_total : 35837 time to create 1 rle with old method : 0.04355597496032715 time for calcul the mask position with numpy : 0.054956912994384766 nb_pixel_total : 125449 time to create 1 rle with old method : 0.14317560195922852 time for calcul the mask position with numpy : 0.055344581604003906 nb_pixel_total : 290457 time to create 1 rle with new method : 1.3957552909851074 time for calcul the mask position with numpy : 0.05752062797546387 nb_pixel_total : 268181 time to create 1 rle with new method : 0.6294033527374268 time for calcul the mask position with numpy : 0.05485725402832031 nb_pixel_total : 34798 time to create 1 rle with old method : 0.04711294174194336 time for calcul the mask position with numpy : 0.06231522560119629 nb_pixel_total : 59578 time to create 1 rle with old method : 0.06995415687561035 time for calcul the mask position with numpy : 0.05526232719421387 nb_pixel_total : 318 time to create 1 rle with old method : 0.0005269050598144531 time for calcul the mask position with numpy : 0.0565648078918457 nb_pixel_total : 87117 time to create 1 rle with old method : 0.10170650482177734 time for calcul the mask position with numpy : 0.05887198448181152 nb_pixel_total : 53728 time to create 1 rle with old method : 0.06471085548400879 create new chi : 39.556543588638306 time to delete rle : 0.08490562438964844 batch 1 Loaded 69 chid ids of type : 4855 Number RLEs to save : 57755 TO DO : save crop sub photo not yet done ! save time : 7.02149224281311 nb_obj : 95 nb_hashtags : 9 time to prepare the origin masks : 80.44348740577698 time for calcul the mask position with numpy : 0.5576200485229492 nb_pixel_total : 4036253 time to create 1 rle with new method : 2.2649059295654297 time for calcul the mask position with numpy : 0.05831098556518555 nb_pixel_total : 250507 time to create 1 rle with new method : 2.099835157394409 time for calcul the mask position with numpy : 0.05416250228881836 nb_pixel_total : 27742 time to create 1 rle with old method : 0.03252148628234863 time for calcul the mask position with numpy : 0.05446743965148926 nb_pixel_total : 115954 time to create 1 rle with old method : 0.1299762725830078 time for calcul the mask position with numpy : 0.05452084541320801 nb_pixel_total : 77113 time to create 1 rle with old method : 0.09271907806396484 time for calcul the mask position with numpy : 0.055139780044555664 nb_pixel_total : 40521 time to create 1 rle with old method : 0.04704093933105469 time for calcul the mask position with numpy : 0.05944514274597168 nb_pixel_total : 300025 time to create 1 rle with new method : 1.8290741443634033 time for calcul the mask position with numpy : 0.06802129745483398 nb_pixel_total : 533193 time to create 1 rle with new method : 0.8208847045898438 time for calcul the mask position with numpy : 0.05749917030334473 nb_pixel_total : 58361 time to create 1 rle with old method : 0.06896853446960449 time for calcul the mask position with numpy : 0.05524277687072754 nb_pixel_total : 66298 time to create 1 rle with old method : 0.07846236228942871 time for calcul the mask position with numpy : 0.05533313751220703 nb_pixel_total : 35171 time to create 1 rle with old method : 0.04094219207763672 time for calcul the mask position with numpy : 0.05449938774108887 nb_pixel_total : 83077 time to create 1 rle with old method : 0.09566211700439453 time for calcul the mask position with numpy : 0.054732322692871094 nb_pixel_total : 369 time to create 1 rle with old method : 0.00072479248046875 time for calcul the mask position with numpy : 0.05407428741455078 nb_pixel_total : 34236 time to create 1 rle with old method : 0.03966879844665527 time for calcul the mask position with numpy : 0.0542454719543457 nb_pixel_total : 81735 time to create 1 rle with old method : 0.10220527648925781 time for calcul the mask position with numpy : 0.05446648597717285 nb_pixel_total : 12058 time to create 1 rle with old method : 0.014181852340698242 time for calcul the mask position with numpy : 0.05433058738708496 nb_pixel_total : 17095 time to create 1 rle with old method : 0.02008366584777832 time for calcul the mask position with numpy : 0.054822683334350586 nb_pixel_total : 66772 time to create 1 rle with old method : 0.07670974731445312 time for calcul the mask position with numpy : 0.054766178131103516 nb_pixel_total : 67230 time to create 1 rle with old method : 0.07790517807006836 time for calcul the mask position with numpy : 0.054325103759765625 nb_pixel_total : 47574 time to create 1 rle with old method : 0.06496763229370117 time for calcul the mask position with numpy : 0.06094861030578613 nb_pixel_total : 3999 time to create 1 rle with old method : 0.005039215087890625 time for calcul the mask position with numpy : 0.05471682548522949 nb_pixel_total : 46859 time to create 1 rle with old method : 0.054198265075683594 time for calcul the mask position with numpy : 0.05550980567932129 nb_pixel_total : 135458 time to create 1 rle with old method : 0.15688705444335938 time for calcul the mask position with numpy : 0.05516839027404785 nb_pixel_total : 47096 time to create 1 rle with old method : 0.054998159408569336 time for calcul the mask position with numpy : 0.0550694465637207 nb_pixel_total : 119605 time to create 1 rle with old method : 0.13757562637329102 time for calcul the mask position with numpy : 0.05868673324584961 nb_pixel_total : 41611 time to create 1 rle with old method : 0.05096745491027832 time for calcul the mask position with numpy : 0.05584001541137695 nb_pixel_total : 176407 time to create 1 rle with new method : 2.077057361602783 time for calcul the mask position with numpy : 0.05554032325744629 nb_pixel_total : 50909 time to create 1 rle with old method : 0.05887794494628906 time for calcul the mask position with numpy : 0.05480599403381348 nb_pixel_total : 30233 time to create 1 rle with old method : 0.034830570220947266 time for calcul the mask position with numpy : 0.055727243423461914 nb_pixel_total : 57353 time to create 1 rle with old method : 0.06756258010864258 time for calcul the mask position with numpy : 0.05830669403076172 nb_pixel_total : 23357 time to create 1 rle with old method : 0.02798771858215332 time for calcul the mask position with numpy : 0.05711793899536133 nb_pixel_total : 15823 time to create 1 rle with old method : 0.019289493560791016 time for calcul the mask position with numpy : 0.05618572235107422 nb_pixel_total : 75255 time to create 1 rle with old method : 0.08641576766967773 time for calcul the mask position with numpy : 0.05886125564575195 nb_pixel_total : 42992 time to create 1 rle with old method : 0.049903154373168945 time for calcul the mask position with numpy : 0.06472182273864746 nb_pixel_total : 112128 time to create 1 rle with old method : 0.12936854362487793 time for calcul the mask position with numpy : 0.05756258964538574 nb_pixel_total : 52953 time to create 1 rle with old method : 0.06630539894104004 time for calcul the mask position with numpy : 0.06253671646118164 nb_pixel_total : 135762 time to create 1 rle with old method : 0.19098663330078125 time for calcul the mask position with numpy : 0.05996513366699219 nb_pixel_total : 63058 time to create 1 rle with old method : 0.07330822944641113 time for calcul the mask position with numpy : 0.06047224998474121 nb_pixel_total : 72151 time to create 1 rle with old method : 0.0878610610961914 time for calcul the mask position with numpy : 0.05411410331726074 nb_pixel_total : 1041 time to create 1 rle with old method : 0.0014832019805908203 time for calcul the mask position with numpy : 0.05726814270019531 nb_pixel_total : 49647 time to create 1 rle with old method : 0.06072568893432617 time for calcul the mask position with numpy : 0.06544137001037598 nb_pixel_total : 79778 time to create 1 rle with old method : 0.09489154815673828 time for calcul the mask position with numpy : 0.08184027671813965 nb_pixel_total : 181260 time to create 1 rle with new method : 2.2521417140960693 time for calcul the mask position with numpy : 0.061551809310913086 nb_pixel_total : 70356 time to create 1 rle with old method : 0.09994912147521973 time for calcul the mask position with numpy : 0.07597064971923828 nb_pixel_total : 258847 time to create 1 rle with new method : 1.3543365001678467 time for calcul the mask position with numpy : 0.08181953430175781 nb_pixel_total : 470821 time to create 1 rle with new method : 1.733506441116333 time for calcul the mask position with numpy : 0.06992721557617188 nb_pixel_total : 44028 time to create 1 rle with old method : 0.05470108985900879 time for calcul the mask position with numpy : 0.06899571418762207 nb_pixel_total : 80414 time to create 1 rle with old method : 0.09415912628173828 time for calcul the mask position with numpy : 0.06354212760925293 nb_pixel_total : 43242 time to create 1 rle with old method : 0.05245566368103027 time for calcul the mask position with numpy : 0.05664706230163574 nb_pixel_total : 2284 time to create 1 rle with old method : 0.0029931068420410156 time for calcul the mask position with numpy : 0.06420421600341797 nb_pixel_total : 94175 time to create 1 rle with old method : 0.10737133026123047 time for calcul the mask position with numpy : 0.07585287094116211 nb_pixel_total : 74784 time to create 1 rle with old method : 0.08979463577270508 time for calcul the mask position with numpy : 0.06058979034423828 nb_pixel_total : 116 time to create 1 rle with old method : 0.0003032684326171875 time for calcul the mask position with numpy : 0.0652170181274414 nb_pixel_total : 2173 time to create 1 rle with old method : 0.002786397933959961 time for calcul the mask position with numpy : 0.0706031322479248 nb_pixel_total : 76296 time to create 1 rle with old method : 0.08767151832580566 time for calcul the mask position with numpy : 0.06791901588439941 nb_pixel_total : 47369 time to create 1 rle with old method : 0.05816817283630371 time for calcul the mask position with numpy : 0.08872818946838379 nb_pixel_total : 308368 time to create 1 rle with new method : 1.674581527709961 time for calcul the mask position with numpy : 0.07197356224060059 nb_pixel_total : 30750 time to create 1 rle with old method : 0.03619098663330078 time for calcul the mask position with numpy : 0.06157350540161133 nb_pixel_total : 5648 time to create 1 rle with old method : 0.008330345153808594 time for calcul the mask position with numpy : 0.06365394592285156 nb_pixel_total : 31185 time to create 1 rle with old method : 0.03624558448791504 time for calcul the mask position with numpy : 0.05917024612426758 nb_pixel_total : 1767 time to create 1 rle with old method : 0.0023071765899658203 time for calcul the mask position with numpy : 0.06576180458068848 nb_pixel_total : 144000 time to create 1 rle with old method : 0.16183018684387207 time for calcul the mask position with numpy : 0.0551905632019043 nb_pixel_total : 762 time to create 1 rle with old method : 0.0013537406921386719 time for calcul the mask position with numpy : 0.062137603759765625 nb_pixel_total : 74157 time to create 1 rle with old method : 0.08721446990966797 time for calcul the mask position with numpy : 0.056748151779174805 nb_pixel_total : 31448 time to create 1 rle with old method : 0.03594255447387695 time for calcul the mask position with numpy : 0.06089377403259277 nb_pixel_total : 73304 time to create 1 rle with old method : 0.08460640907287598 time for calcul the mask position with numpy : 0.05602216720581055 nb_pixel_total : 8878 time to create 1 rle with old method : 0.010645389556884766 time for calcul the mask position with numpy : 0.05601191520690918 nb_pixel_total : 42156 time to create 1 rle with old method : 0.04829692840576172 time for calcul the mask position with numpy : 0.05842304229736328 nb_pixel_total : 49286 time to create 1 rle with old method : 0.05821967124938965 time for calcul the mask position with numpy : 0.059854984283447266 nb_pixel_total : 77998 time to create 1 rle with old method : 0.09399175643920898 time for calcul the mask position with numpy : 0.055504560470581055 nb_pixel_total : 39418 time to create 1 rle with old method : 0.04480171203613281 time for calcul the mask position with numpy : 0.05796003341674805 nb_pixel_total : 60868 time to create 1 rle with old method : 0.08298659324645996 time for calcul the mask position with numpy : 0.06509256362915039 nb_pixel_total : 220603 time to create 1 rle with new method : 1.3266074657440186 time for calcul the mask position with numpy : 0.06747961044311523 nb_pixel_total : 186061 time to create 1 rle with new method : 1.9471924304962158 time for calcul the mask position with numpy : 0.06076979637145996 nb_pixel_total : 53537 time to create 1 rle with old method : 0.07871723175048828 time for calcul the mask position with numpy : 0.06422805786132812 nb_pixel_total : 158813 time to create 1 rle with new method : 1.5072600841522217 time for calcul the mask position with numpy : 0.06502270698547363 nb_pixel_total : 215257 time to create 1 rle with new method : 1.6849851608276367 time for calcul the mask position with numpy : 0.06385111808776855 nb_pixel_total : 186158 time to create 1 rle with new method : 1.2301006317138672 time for calcul the mask position with numpy : 0.06025815010070801 nb_pixel_total : 31162 time to create 1 rle with old method : 0.03628706932067871 time for calcul the mask position with numpy : 0.05309939384460449 nb_pixel_total : 12059 time to create 1 rle with old method : 0.013753414154052734 time for calcul the mask position with numpy : 0.0582890510559082 nb_pixel_total : 31745 time to create 1 rle with old method : 0.03658699989318848 time for calcul the mask position with numpy : 0.07066178321838379 nb_pixel_total : 556800 time to create 1 rle with new method : 1.4381766319274902 time for calcul the mask position with numpy : 0.05743575096130371 nb_pixel_total : 16231 time to create 1 rle with old method : 0.019892454147338867 time for calcul the mask position with numpy : 0.05577373504638672 nb_pixel_total : 11052 time to create 1 rle with old method : 0.01326894760131836 time for calcul the mask position with numpy : 0.05698084831237793 nb_pixel_total : 43280 time to create 1 rle with old method : 0.0557253360748291 time for calcul the mask position with numpy : 0.06376981735229492 nb_pixel_total : 39373 time to create 1 rle with old method : 0.047035932540893555 time for calcul the mask position with numpy : 0.05783653259277344 nb_pixel_total : 49788 time to create 1 rle with old method : 0.05864882469177246 time for calcul the mask position with numpy : 0.06333184242248535 nb_pixel_total : 221141 time to create 1 rle with new method : 2.0211775302886963 time for calcul the mask position with numpy : 0.07662677764892578 nb_pixel_total : 632986 time to create 1 rle with new method : 1.8422982692718506 time for calcul the mask position with numpy : 0.06435966491699219 nb_pixel_total : 42429 time to create 1 rle with old method : 0.049920082092285156 time for calcul the mask position with numpy : 0.07908964157104492 nb_pixel_total : 187573 time to create 1 rle with new method : 1.375732183456421 time for calcul the mask position with numpy : 0.06808590888977051 nb_pixel_total : 189578 time to create 1 rle with new method : 1.5369064807891846 time for calcul the mask position with numpy : 0.061272621154785156 nb_pixel_total : 4332 time to create 1 rle with old method : 0.005612373352050781 time for calcul the mask position with numpy : 0.05550503730773926 nb_pixel_total : 74619 time to create 1 rle with old method : 0.08369326591491699 time for calcul the mask position with numpy : 0.053613901138305664 nb_pixel_total : 2587 time to create 1 rle with old method : 0.002872467041015625 time for calcul the mask position with numpy : 0.05362677574157715 nb_pixel_total : 1119 time to create 1 rle with old method : 0.0022089481353759766 create new chi : 43.84829664230347 time to delete rle : 0.014390945434570312 batch 1 Loaded 96 chid ids of type : 4855 Number RLEs to save : 70133 TO DO : save crop sub photo not yet done ! save time : 13.02781367301941 nb_obj : 69 nb_hashtags : 8 time to prepare the origin masks : 65.13797903060913 time for calcul the mask position with numpy : 0.44318222999572754 nb_pixel_total : 4575831 time to create 1 rle with new method : 2.9671630859375 time for calcul the mask position with numpy : 0.06563782691955566 nb_pixel_total : 375187 time to create 1 rle with new method : 1.363691806793213 time for calcul the mask position with numpy : 0.05341672897338867 nb_pixel_total : 39455 time to create 1 rle with old method : 0.04471230506896973 time for calcul the mask position with numpy : 0.05370831489562988 nb_pixel_total : 68931 time to create 1 rle with old method : 0.08119487762451172 time for calcul the mask position with numpy : 0.05194520950317383 nb_pixel_total : 19280 time to create 1 rle with old method : 0.021547794342041016 time for calcul the mask position with numpy : 0.05168747901916504 nb_pixel_total : 95605 time to create 1 rle with old method : 0.10271191596984863 time for calcul the mask position with numpy : 0.051447153091430664 nb_pixel_total : 16661 time to create 1 rle with old method : 0.018505573272705078 time for calcul the mask position with numpy : 0.0538637638092041 nb_pixel_total : 132733 time to create 1 rle with old method : 0.1532273292541504 time for calcul the mask position with numpy : 0.052002906799316406 nb_pixel_total : 821 time to create 1 rle with old method : 0.001004934310913086 time for calcul the mask position with numpy : 0.05575823783874512 nb_pixel_total : 243995 time to create 1 rle with new method : 1.9613423347473145 time for calcul the mask position with numpy : 0.054065704345703125 nb_pixel_total : 122102 time to create 1 rle with old method : 0.14553046226501465 time for calcul the mask position with numpy : 0.05243992805480957 nb_pixel_total : 16929 time to create 1 rle with old method : 0.018822669982910156 time for calcul the mask position with numpy : 0.054811954498291016 nb_pixel_total : 181865 time to create 1 rle with new method : 1.5040881633758545 time for calcul the mask position with numpy : 0.050014495849609375 nb_pixel_total : 68277 time to create 1 rle with old method : 0.07301497459411621 time for calcul the mask position with numpy : 0.05226325988769531 nb_pixel_total : 196331 time to create 1 rle with new method : 1.2105398178100586 time for calcul the mask position with numpy : 0.05116605758666992 nb_pixel_total : 5214 time to create 1 rle with old method : 0.005975961685180664 time for calcul the mask position with numpy : 0.05278897285461426 nb_pixel_total : 28438 time to create 1 rle with old method : 0.03136444091796875 time for calcul the mask position with numpy : 0.05232954025268555 nb_pixel_total : 900 time to create 1 rle with old method : 0.0012612342834472656 time for calcul the mask position with numpy : 0.05220341682434082 nb_pixel_total : 66640 time to create 1 rle with old method : 0.07513952255249023 time for calcul the mask position with numpy : 0.05471181869506836 nb_pixel_total : 75484 time to create 1 rle with old method : 0.08914685249328613 time for calcul the mask position with numpy : 0.056542158126831055 nb_pixel_total : 116129 time to create 1 rle with old method : 0.14792346954345703 time for calcul the mask position with numpy : 0.05445146560668945 nb_pixel_total : 23429 time to create 1 rle with old method : 0.028725147247314453 time for calcul the mask position with numpy : 0.05578804016113281 nb_pixel_total : 172889 time to create 1 rle with new method : 1.4960300922393799 time for calcul the mask position with numpy : 0.05418729782104492 nb_pixel_total : 10535 time to create 1 rle with old method : 0.012949705123901367 time for calcul the mask position with numpy : 0.05541205406188965 nb_pixel_total : 16938 time to create 1 rle with old method : 0.020308494567871094 time for calcul the mask position with numpy : 0.055520057678222656 nb_pixel_total : 148967 time to create 1 rle with old method : 0.17049050331115723 time for calcul the mask position with numpy : 0.0563201904296875 nb_pixel_total : 160514 time to create 1 rle with new method : 2.280855894088745 time for calcul the mask position with numpy : 0.05411505699157715 nb_pixel_total : 17588 time to create 1 rle with old method : 0.020220041275024414 time for calcul the mask position with numpy : 0.054726362228393555 nb_pixel_total : 326011 time to create 1 rle with new method : 1.2884809970855713 time for calcul the mask position with numpy : 0.05649518966674805 nb_pixel_total : 1629 time to create 1 rle with old method : 0.002074003219604492 time for calcul the mask position with numpy : 0.054347991943359375 nb_pixel_total : 123648 time to create 1 rle with old method : 0.16486859321594238 time for calcul the mask position with numpy : 0.055469512939453125 nb_pixel_total : 2186 time to create 1 rle with old method : 0.0027418136596679688 time for calcul the mask position with numpy : 0.05348491668701172 nb_pixel_total : 55105 time to create 1 rle with old method : 0.0662684440612793 time for calcul the mask position with numpy : 0.05434250831604004 nb_pixel_total : 142990 time to create 1 rle with old method : 0.17437386512756348 time for calcul the mask position with numpy : 0.05563235282897949 nb_pixel_total : 15152 time to create 1 rle with old method : 0.01881718635559082 time for calcul the mask position with numpy : 0.052800893783569336 nb_pixel_total : 118079 time to create 1 rle with old method : 0.225571870803833 time for calcul the mask position with numpy : 0.07387399673461914 nb_pixel_total : 73283 time to create 1 rle with old method : 0.10089945793151855 time for calcul the mask position with numpy : 0.05593609809875488 nb_pixel_total : 29894 time to create 1 rle with old method : 0.03522062301635742 time for calcul the mask position with numpy : 0.05843234062194824 nb_pixel_total : 159133 time to create 1 rle with new method : 2.6164169311523438 time for calcul the mask position with numpy : 0.061341047286987305 nb_pixel_total : 9744 time to create 1 rle with old method : 0.015941381454467773 time for calcul the mask position with numpy : 0.058101654052734375 nb_pixel_total : 221279 time to create 1 rle with new method : 1.8014740943908691 time for calcul the mask position with numpy : 0.05409669876098633 nb_pixel_total : 92932 time to create 1 rle with old method : 0.10662508010864258 time for calcul the mask position with numpy : 0.05599212646484375 nb_pixel_total : 2293 time to create 1 rle with old method : 0.0037810802459716797 time for calcul the mask position with numpy : 0.3587074279785156 nb_pixel_total : 702469 time to create 1 rle with new method : 2.0210766792297363 time for calcul the mask position with numpy : 0.05421113967895508 nb_pixel_total : 41047 time to create 1 rle with old method : 0.047895193099975586 time for calcul the mask position with numpy : 0.053969383239746094 nb_pixel_total : 3445 time to create 1 rle with old method : 0.0043735504150390625 time for calcul the mask position with numpy : 0.05454230308532715 nb_pixel_total : 31984 time to create 1 rle with old method : 0.037676334381103516 time for calcul the mask position with numpy : 0.05855989456176758 nb_pixel_total : 188753 time to create 1 rle with new method : 1.91007661819458 time for calcul the mask position with numpy : 0.061789512634277344 nb_pixel_total : 537342 time to create 1 rle with new method : 1.7937417030334473 time for calcul the mask position with numpy : 0.0544123649597168 nb_pixel_total : 30474 time to create 1 rle with old method : 0.03698253631591797 time for calcul the mask position with numpy : 0.05455160140991211 nb_pixel_total : 109360 time to create 1 rle with old method : 0.1275327205657959 time for calcul the mask position with numpy : 0.0549769401550293 nb_pixel_total : 156518 time to create 1 rle with new method : 2.080599069595337 time for calcul the mask position with numpy : 0.05556607246398926 nb_pixel_total : 28798 time to create 1 rle with old method : 0.03619956970214844 time for calcul the mask position with numpy : 0.06807518005371094 nb_pixel_total : 864430 time to create 1 rle with new method : 1.5095045566558838 time for calcul the mask position with numpy : 0.05412650108337402 nb_pixel_total : 2617 time to create 1 rle with old method : 0.004751920700073242 time for calcul the mask position with numpy : 0.06439852714538574 nb_pixel_total : 772683 time to create 1 rle with new method : 1.617645502090454 time for calcul the mask position with numpy : 0.05425214767456055 nb_pixel_total : 4591 time to create 1 rle with old method : 0.0056803226470947266 time for calcul the mask position with numpy : 0.10330319404602051 nb_pixel_total : 14734 time to create 1 rle with old method : 0.03493070602416992 time for calcul the mask position with numpy : 0.05640864372253418 nb_pixel_total : 23979 time to create 1 rle with old method : 0.029087543487548828 time for calcul the mask position with numpy : 0.053841590881347656 nb_pixel_total : 4214 time to create 1 rle with old method : 0.005754709243774414 time for calcul the mask position with numpy : 0.059302568435668945 nb_pixel_total : 8337 time to create 1 rle with old method : 0.00975942611694336 time for calcul the mask position with numpy : 0.05913186073303223 nb_pixel_total : 554878 time to create 1 rle with new method : 1.741621732711792 time for calcul the mask position with numpy : 0.053896188735961914 nb_pixel_total : 20907 time to create 1 rle with old method : 0.02543497085571289 time for calcul the mask position with numpy : 0.054672956466674805 nb_pixel_total : 115427 time to create 1 rle with old method : 0.1380760669708252 time for calcul the mask position with numpy : 0.05466341972351074 nb_pixel_total : 27904 time to create 1 rle with old method : 0.03464198112487793 time for calcul the mask position with numpy : 0.05742144584655762 nb_pixel_total : 194788 time to create 1 rle with new method : 1.5597383975982666 time for calcul the mask position with numpy : 0.06083965301513672 nb_pixel_total : 77367 time to create 1 rle with old method : 0.10355472564697266 time for calcul the mask position with numpy : 0.05617880821228027 nb_pixel_total : 25230 time to create 1 rle with old method : 0.030836105346679688 time for calcul the mask position with numpy : 0.0543065071105957 nb_pixel_total : 38492 time to create 1 rle with old method : 0.04555392265319824 time for calcul the mask position with numpy : 0.06290602684020996 nb_pixel_total : 27406 time to create 1 rle with old method : 0.03536105155944824 create new chi : 41.25058197975159 time to delete rle : 0.008281707763671875 batch 1 Loaded 70 chid ids of type : 4855 Number RLEs to save : 52698 TO DO : save crop sub photo not yet done ! save time : 4.264673709869385 nb_obj : 8 nb_hashtags : 5 time to prepare the origin masks : 28.59167218208313 time for calcul the mask position with numpy : 0.5521104335784912 nb_pixel_total : 2299988 time to create 1 rle with new method : 2.0382325649261475 time for calcul the mask position with numpy : 0.07625055313110352 nb_pixel_total : 1962 time to create 1 rle with old method : 0.0025844573974609375 time for calcul the mask position with numpy : 0.18819761276245117 nb_pixel_total : 467607 time to create 1 rle with new method : 1.149064540863037 time for calcul the mask position with numpy : 0.07732200622558594 nb_pixel_total : 122312 time to create 1 rle with old method : 0.2107996940612793 time for calcul the mask position with numpy : 0.06119799613952637 nb_pixel_total : 74543 time to create 1 rle with old method : 0.08663153648376465 time for calcul the mask position with numpy : 0.06803250312805176 nb_pixel_total : 4551 time to create 1 rle with old method : 0.00540614128112793 time for calcul the mask position with numpy : 0.08574843406677246 nb_pixel_total : 251928 time to create 1 rle with new method : 1.5363199710845947 time for calcul the mask position with numpy : 0.2216966152191162 nb_pixel_total : 2372642 time to create 1 rle with new method : 1.0897674560546875 time for calcul the mask position with numpy : 0.8339190483093262 nb_pixel_total : 7383667 time to create 1 rle with new method : 1.418365478515625 create new chi : 9.998584747314453 time to delete rle : 0.008278846740722656 batch 1 Loaded 9 chid ids of type : 4855 Number RLEs to save : 20157 TO DO : save crop sub photo not yet done ! save time : 5.579636096954346 map_output_result : {1342523624: (0.48893964170942017, 'Should be the crop_list due to order', 0.27234287453252337), 1342523598: (0.48893964170942017, 'Should be the crop_list due to order', 0.4341454085637542), 1342523593: (0.48893964170942017, 'Should be the crop_list due to order', 0.36112549903977176), 1342523589: (0.48893964170942017, 'Should be the crop_list due to order', 0.8881447847016314)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1342523624, 1342523598, 1342523593, 1342523589] Looping around the photos to save general results len do output : 4 /1342523624.Didn't retrieve data . /1342523598.Didn't retrieve data . /1342523593.Didn't retrieve data . /1342523589.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4741', None, None, None, None, None, None, None, '2654458') ('4741', '21222676', '1342523624', None, None, None, None, None, '2654458') ('4741', None, None, None, None, None, None, None, '2654458') ('4741', '21222676', '1342523598', None, None, None, None, None, '2654458') ('4741', None, None, None, None, None, None, None, '2654458') ('4741', '21222676', '1342523593', None, None, None, None, None, '2654458') ('4741', None, None, None, None, None, None, None, '2654458') ('4741', '21222676', '1342523589', None, None, None, None, None, '2654458') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 12 time used for this insertion : 0.03796124458312988 save_final save missing photos in datou_result : time spend for datou_step_exec : 396.436576128006 time spend to save output : 0.043207406997680664 total time spend for step 5 : 396.47978353500366 step6:crop_condition Mon Mar 10 20:11:13 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 4855 Loading chi in step crop for list_pids : 4 ! batch 1 Loaded 244 chid ids of type : 4855 begin to crop the class : papier param for this class : {'min_score': 0.1} filtre for class : papier hashtag_id of this class : 492668766 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 25 About to insert : list_path_to_insert length 25 new photo from crops ! About to upload 25 photos upload in portfolio : 3015255 init cache_photo without model_param we have 25 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741633905_471565 we have uploaded 25 photos in the portfolio 3015255 time of upload the photos Elapsed time : 11.095356225967407 we have finished the crop for the class : papier begin to crop the class : carton param for this class : {'min_score': 0.1} filtre for class : carton hashtag_id of this class : 492774966 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 60 About to insert : list_path_to_insert length 60 new photo from crops ! About to upload 60 photos upload in portfolio : 3015255 init cache_photo without model_param we have 60 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741633954_471565 we have uploaded 60 photos in the portfolio 3015255 time of upload the photos Elapsed time : 22.371483325958252 we have finished the crop for the class : carton begin to crop the class : metal param for this class : {'min_score': 0.1} filtre for class : metal hashtag_id of this class : 492628673 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 16 About to insert : list_path_to_insert length 16 new photo from crops ! About to upload 16 photos upload in portfolio : 3015255 init cache_photo without model_param we have 16 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741633983_471565 we have uploaded 16 photos in the portfolio 3015255 time of upload the photos Elapsed time : 5.2475244998931885 we have finished the crop for the class : metal begin to crop the class : pet_clair param for this class : {'min_score': 0.1} filtre for class : pet_clair hashtag_id of this class : 2107755846 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 3015255 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741633993_471565 we have uploaded 6 photos in the portfolio 3015255 time of upload the photos Elapsed time : 1.908064842224121 we have finished the crop for the class : pet_clair begin to crop the class : pehd param for this class : {'min_score': 0.1} filtre for class : pehd hashtag_id of this class : 628944319 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 3015255 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741633998_471565 we have uploaded 1 photos in the portfolio 3015255 time of upload the photos Elapsed time : 2.460331439971924 we have finished the crop for the class : pehd begin to crop the class : pet_fonce param for this class : {'min_score': 0.1} filtre for class : pet_fonce hashtag_id of this class : 2107755900 begin to crop the class : sac param for this class : {'min_score': 0.1} filtre for class : sac hashtag_id of this class : 492618350 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 17 About to insert : list_path_to_insert length 17 new photo from crops ! About to upload 17 photos upload in portfolio : 3015255 init cache_photo without model_param we have 17 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741634010_471565 we have uploaded 17 photos in the portfolio 3015255 time of upload the photos Elapsed time : 11.211219549179077 we have finished the crop for the class : sac begin to crop the class : textiles param for this class : {'min_score': 0.1} filtre for class : textiles hashtag_id of this class : 494789423 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 3015255 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741634023_471565 we have uploaded 2 photos in the portfolio 3015255 time of upload the photos Elapsed time : 1.0806684494018555 we have finished the crop for the class : textiles begin to crop the class : verre param for this class : {'min_score': 0.1} filtre for class : verre hashtag_id of this class : 492631919 begin to crop the class : organique param for this class : {'min_score': 0.1} filtre for class : organique hashtag_id of this class : 530726390 begin to crop the class : dasri param for this class : {'min_score': 0.1} filtre for class : dasri hashtag_id of this class : 1608424396 begin to crop the class : masque param for this class : {'min_score': 0.1} filtre for class : masque hashtag_id of this class : 492626781 begin to crop the class : encombrant param for this class : {'min_score': 0.1} filtre for class : encombrant hashtag_id of this class : 605975549 begin to crop the class : autre param for this class : {'min_score': 0.1} filtre for class : autre hashtag_id of this class : 494826614 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 30 About to insert : list_path_to_insert length 30 new photo from crops ! About to upload 30 photos upload in portfolio : 3015255 init cache_photo without model_param we have 30 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741634057_471565 we have uploaded 30 photos in the portfolio 3015255 time of upload the photos Elapsed time : 13.342741966247559 we have finished the crop for the class : autre begin to crop the class : autre_emballage param for this class : {'min_score': 0.1} filtre for class : autre_emballage hashtag_id of this class : 2107760328 map_result returned by crop_photo_return_map_crop : length : 0 About to insert : list_path_to_insert length 0 new photo from crops ! About to upload 0 photos WARNING : list_path_to_insert is empty, cannot upload ! we have finished the crop for the class : autre_emballage begin to crop the class : autre_non_emballage param for this class : {'min_score': 0.1} filtre for class : autre_non_emballage hashtag_id of this class : 2107760329 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 3015255 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741634076_471565 we have uploaded 6 photos in the portfolio 3015255 time of upload the photos Elapsed time : 2.6016671657562256 we have finished the crop for the class : autre_non_emballage begin to crop the class : environnement param for this class : {'min_score': 0.1} filtre for class : environnement hashtag_id of this class : 493012381 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 28 About to insert : list_path_to_insert length 28 new photo from crops ! About to upload 28 photos upload in portfolio : 3015255 init cache_photo without model_param we have 28 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1741634194_471565 we have uploaded 28 photos in the portfolio 3015255 time of upload the photos Elapsed time : 17.92669653892517 we have finished the crop for the class : environnement delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1342523624, 1342523598, 1342523593, 1342523589] Looping around the photos to save general results len do output : 191 /1342643908Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342643910Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342643913Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342643915Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342643917Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342643919Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342643921Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342643923Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342643925Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342643929Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342643931Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342643933Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342643935Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342643937Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342643939Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342643941Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342643943Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342643945Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342643947Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342643949Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342643951Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342643953Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342643955Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342643957Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342643959Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644283Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644285Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644287Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644289Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644291Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644292Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644293Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644294Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644295Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644296Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644297Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644298Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644299Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644300Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644301Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644302Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644304Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644305Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644306Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644307Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644308Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644309Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644310Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644311Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644312Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644313Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644314Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644315Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644316Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644318Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644321Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644323Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644325Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644327Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644328Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644329Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644330Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644331Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644332Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644334Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644335Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644336Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644337Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644338Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644339Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644342Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644345Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644348Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644350Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644352Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644355Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644357Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644359Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644364Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644366Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644368Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644370Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644371Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644372Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644373Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644378Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644379Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644380Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644382Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644383Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644384Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644386Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644388Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644390Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644392Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644393Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644394Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644395Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644396Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644397Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644398Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644411Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644412Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644413Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644414Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644416Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644417Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644441Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644482Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644483Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644484Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644485Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644486Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644487Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644488Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644490Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644498Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644499Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644503Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644505Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644506Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644507Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644508Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644509Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644511Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644526Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644528Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644575Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644576Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644577Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644578Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644579Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644580Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644586Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644588Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644589Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644590Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644591Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644592Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644593Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644594Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644595Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644596Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644597Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644598Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644600Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644601Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644602Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644603Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644604Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644605Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644606Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644607Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644608Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644609Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644610Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644612Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644615Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644616Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644621Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644622Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644623Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342644625Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645232Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645242Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645249Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645259Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645269Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645279Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645289Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645304Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645316Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645320Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645332Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645344Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645348Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645356Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645360Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645369Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645370Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645374Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645375Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645379Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645380Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645384Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645386Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645389Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645392Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645394Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645397Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1342645399Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4741', None, None, None, None, None, None, None, '2654458') ('4741', '21222676', '1342523624', None, None, None, None, None, '2654458') ('4741', None, None, None, None, None, None, None, '2654458') ('4741', '21222676', '1342523598', None, None, None, None, None, '2654458') ('4741', None, None, None, None, None, None, None, '2654458') ('4741', '21222676', '1342523593', None, None, None, None, None, '2654458') ('4741', None, None, None, None, None, None, None, '2654458') ('4741', '21222676', '1342523589', None, None, None, None, None, '2654458') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 577 time used for this insertion : 0.8109285831451416 save_final save missing photos in datou_result : time spend for datou_step_exec : 343.35047340393066 time spend to save output : 0.9479668140411377 total time spend for step 6 : 344.2984402179718 step7:ventilate_hashtags_in_portfolio Mon Mar 10 20:16:57 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! Iterating over portfolio : 21222676 get user id for portfolio 21222676 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21222676 AND mptpi.`type`=4855 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','carton','metal','pet_clair','pehd','pet_fonce','pet_opaque','barquette_opaque','film_plastique','ela','sac','textiles','verre','organique','dasri','masque','encombrant','autre_emballage','autre_non_emballage','environnement','autre','flux_dev','mal_croppe','flou','film_dev_souple','sac_om_plein')) AND mptpi.`min_score`=0.1 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21222676 AND mptpi.`type`=4855 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','carton','metal','pet_clair','pehd','pet_fonce','pet_opaque','barquette_opaque','film_plastique','ela','sac','textiles','verre','organique','dasri','masque','encombrant','autre_emballage','autre_non_emballage','environnement','autre','flux_dev','mal_croppe','flou','film_dev_souple','sac_om_plein')) AND mptpi.`min_score`=0.1 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21222676 AND mptpi.`type`=4856 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','carton','metal','pet_clair','pehd','pet_fonce','pet_opaque','barquette_opaque','film_plastique','ela','sac','textiles','verre','organique','dasri','masque','encombrant','autre_emballage','autre_non_emballage','environnement','autre','flux_dev','mal_croppe','flou','film_dev_souple','sac_om_plein')) AND mptpi.`min_score`=0.1 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21222676 AND mptpi.`type`=4856 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','carton','metal','pet_clair','pehd','pet_fonce','pet_opaque','barquette_opaque','film_plastique','ela','sac','textiles','verre','organique','dasri','masque','encombrant','autre_emballage','autre_non_emballage','environnement','autre','flux_dev','mal_croppe','flou','film_dev_souple','sac_om_plein')) AND mptpi.`min_score`=0.1 To do lien utilise dans velours : https://www.fotonower.com/velours/21238826,21238827,21238828,21238829,21238830,21238835,21238837,21238838,21238839,21238840,21238841,21238842,21238844,21238845,21238846,21238847,21238848,21238849,21238850,21238851?tags=papier,carton,metal,pet_clair,pehd,ela,textiles,verre,organique,dasri,masque,encombrant,autre_non_emballage,environnement,autre,flux_dev,mal_croppe,flou,film_dev_souple,sac_om_plein&datou_id_consolidate=4742&port_consolidate=21222676 Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1342523624, 1342523598, 1342523593, 1342523589] Looping around the photos to save general results len do output : 1 /21222676. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4741', None, None, None, None, None, None, None, '2654458') ('4741', '21222676', '1342523624', None, None, None, None, None, '2654458') ('4741', None, None, None, None, None, None, None, '2654458') ('4741', '21222676', '1342523598', None, None, None, None, None, '2654458') ('4741', None, None, None, None, None, None, None, '2654458') ('4741', '21222676', '1342523593', None, None, None, None, None, '2654458') ('4741', None, None, None, None, None, None, None, '2654458') ('4741', '21222676', '1342523589', None, None, None, None, None, '2654458') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 5 time used for this insertion : 0.0429537296295166 save_final save missing photos in datou_result : time spend for datou_step_exec : 15.929943084716797 time spend to save output : 0.043267250061035156 total time spend for step 7 : 15.973210334777832 step8:final Mon Mar 10 20:17:13 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1342523624: ('0.37315928113922087',), 1342523598: ('0.37315928113922087',), 1342523593: ('0.37315928113922087',), 1342523589: ('0.37315928113922087',)} new output for save of step final : {1342523624: ('0.37315928113922087',), 1342523598: ('0.37315928113922087',), 1342523593: ('0.37315928113922087',), 1342523589: ('0.37315928113922087',)} [1342523624, 1342523598, 1342523593, 1342523589] Looping around the photos to save general results len do output : 4 /1342523624.Didn't retrieve data . /1342523598.Didn't retrieve data . /1342523593.Didn't retrieve data . /1342523589.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('4741', None, None, None, None, None, None, None, '2654458') ('4741', '21222676', '1342523624', None, None, None, None, None, '2654458') ('4741', None, None, None, None, None, None, None, '2654458') ('4741', '21222676', '1342523598', None, None, None, None, None, '2654458') ('4741', None, None, None, None, None, None, None, '2654458') ('4741', '21222676', '1342523593', None, None, None, None, None, '2654458') ('4741', None, None, None, None, None, None, None, '2654458') ('4741', '21222676', '1342523589', None, None, None, None, None, '2654458') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 12 time used for this insertion : 0.18846511840820312 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.3660006523132324 time spend to save output : 0.1890556812286377 total time spend for step 8 : 0.5550563335418701 step9:velours_tree Mon Mar 10 20:17:14 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 2.365469217300415 time spend to save output : 6.747245788574219e-05 total time spend for step 9 : 2.365536689758301 step10:send_mail_cod Mon Mar 10 20:17:16 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 complete output_args for input 1 complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P21222676_10-03-2025_20_17_16.pdf 21238800 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette212388001741634236 21238801 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette212388011741634238 21238802 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette212388021741634240 21238803 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette212388031741634241 21238804 change filename to text .imagette212388041741634242 21238805 imagette212388051741634242 21238806 imagette212388061741634242 21238807 imagette212388071741634242 21238808 imagette212388081741634242 21238809 imagette212388091741634242 21238810 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette212388101741634242 21238811 change filename to text .change filename to text .imagette212388111741634243 21238812 imagette212388121741634243 21238813 imagette212388131741634243 21238814 imagette212388141741634243 21238815 imagette212388151741634243 21238816 imagette212388161741634243 21238817 imagette212388171741634243 21238818 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette212388181741634243 21238820 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette212388201741634244 21238821 imagette212388211741634246 21238822 imagette212388221741634246 21238823 imagette212388231741634246 21238824 imagette212388241741634246 21238825 imagette212388251741634246 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=21222676 and hashtag_type = 4855 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/21238826,21238827,21238828,21238829,21238830,21238835,21238837,21238838,21238839,21238840,21238841,21238842,21238844,21238845,21238846,21238847,21238848,21238849,21238850,21238851?tags=papier,carton,metal,pet_clair,pehd,ela,textiles,verre,organique,dasri,masque,encombrant,autre_non_emballage,environnement,autre,flux_dev,mal_croppe,flou,film_dev_souple,sac_om_plein&datou_id_consolidate=4742&port_consolidate=21222676 your option no_mail is active, we will not send the real mail to your client args[1342523624] : ((1342523624, 2014.5916170730059, 492609224), (1342523624, -0.3007075378494067, 496442774), '0.37315928113922087') We are sending mail with results at report@fotonower.com,cod@fotonower.com args[1342523598] : ((1342523598, 2073.371732719602, 492609224), (1342523598, -0.2647607599792512, 496442774), '0.37315928113922087') We are sending mail with results at report@fotonower.com,cod@fotonower.com args[1342523593] : ((1342523593, 3981.40409470297, 492609224), (1342523593, -0.3507944862834815, 496442774), '0.37315928113922087') We are sending mail with results at report@fotonower.com,cod@fotonower.com args[1342523589] : ((1342523589, 448.5099047918387, 492688767), (1342523589, -0.2656268576693532, 496442774), '0.37315928113922087') We are sending mail with results at report@fotonower.com,cod@fotonower.com refus_total : 0.37315928113922087 2022-04-13 10:29:59 0 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21222676_10-03-2025_20_17_16.pdf results_Auto_P21222676_10-03-2025_20_17_16.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21222676_10-03-2025_20_17_16.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('4741','21222676','results_Auto_P21222676_10-03-2025_20_17_16.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21222676_10-03-2025_20_17_16.pdf','pdf','','2.46','0.37315928113922087') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1342523624, 1342523598, 1342523593, 1342523589] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('4741', None, None, None, None, None, None, None, '2654458') ('4741', '21222676', '1342523624', None, None, None, None, None, '2654458') ('4741', None, None, None, None, None, None, None, '2654458') ('4741', '21222676', '1342523598', None, None, None, None, None, '2654458') ('4741', None, None, None, None, None, None, None, '2654458') ('4741', '21222676', '1342523593', None, None, None, None, None, '2654458') ('4741', None, None, None, None, None, None, None, '2654458') ('4741', '21222676', '1342523589', None, None, None, None, None, '2654458') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 4 time used for this insertion : 0.0403749942779541 save_final save missing photos in datou_result : time spend for datou_step_exec : 12.892016649246216 time spend to save output : 0.04078555107116699 total time spend for step 10 : 12.932802200317383 step11:split_time_score Mon Mar 10 20:17:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('07', 1), ('09', 3)) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 04022025 21222676 Nombre de photos uploadées : 4 / 23040 (0%) 04022025 21222676 Nombre de photos taguées (types de déchets): 0 / 4 (0%) 04022025 21222676 Nombre de photos taguées (volume) : 0 / 4 (0%) elapsed_time : load_data_split_time_score 2.6226043701171875e-06 elapsed_time : order_list_meta_photo_and_scores 6.67572021484375e-06 ???? elapsed_time : fill_and_build_computed_from_old_data 0.00024390220642089844 elapsed_time : insert_dashboard_record_day_entry 0.08840227127075195 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.41441132458681595 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21222675_10-03-2025_15_19_18.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21222675 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 13156 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 13160 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 13160 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 13157 crop_condition is not consistent : 4 used against 2 in the step definition ! WARNING : number of outputs for step 13157 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 13161 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 13159 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 13159 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 13156 doesn't seem to be define in the database( WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 13161 doesn't seem to be define in the database( WARNING : type of input 3 of step 13159 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13163 have datatype=6 whereas input 2 of step 13157 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13162 have datatype=6 whereas input 2 of step 13157 have datatype=None We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 13161 have datatype=10 whereas input 3 of step 13164 have datatype=6 WARNING : output 0 of step 13161 have datatype=10 whereas input 0 of step 13166 have datatype=18 WARNING : type of input 5 of step 13164 doesn't seem to be define in the database( WARNING : output 0 of step 13166 have datatype=11 whereas input 5 of step 13164 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 1 of step 13158 have datatype=7 whereas input 2 of step 13157 have datatype=None WARNING : type of output 3 of step 13157 doesn't seem to be define in the database( WARNING : type of input 3 of step 13161 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13163 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13162 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21222675 AND mptpi.`type`=4855 To do Qualite : 0.37315928113922087 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21222676_10-03-2025_20_17_16.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21222676 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 13156 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 13160 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 13160 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 13157 crop_condition is not consistent : 4 used against 2 in the step definition ! WARNING : number of outputs for step 13157 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 13161 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 13159 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 13159 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 13156 doesn't seem to be define in the database( WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 13161 doesn't seem to be define in the database( WARNING : type of input 3 of step 13159 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13163 have datatype=6 whereas input 2 of step 13157 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13162 have datatype=6 whereas input 2 of step 13157 have datatype=None We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 13161 have datatype=10 whereas input 3 of step 13164 have datatype=6 WARNING : output 0 of step 13161 have datatype=10 whereas input 0 of step 13166 have datatype=18 WARNING : type of input 5 of step 13164 doesn't seem to be define in the database( WARNING : output 0 of step 13166 have datatype=11 whereas input 5 of step 13164 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 1 of step 13158 have datatype=7 whereas input 2 of step 13157 have datatype=None WARNING : type of output 3 of step 13157 doesn't seem to be define in the database( WARNING : type of input 3 of step 13161 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13163 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13162 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21222676 AND mptpi.`type`=4855 To do Qualite : 0.27154147259571604 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21222677_10-03-2025_19_52_19.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21222677 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 13156 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 13160 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 13160 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 13157 crop_condition is not consistent : 4 used against 2 in the step definition ! WARNING : number of outputs for step 13157 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 13161 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 13159 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 13159 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 13156 doesn't seem to be define in the database( WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 13161 doesn't seem to be define in the database( WARNING : type of input 3 of step 13159 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13163 have datatype=6 whereas input 2 of step 13157 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13162 have datatype=6 whereas input 2 of step 13157 have datatype=None We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 13161 have datatype=10 whereas input 3 of step 13164 have datatype=6 WARNING : output 0 of step 13161 have datatype=10 whereas input 0 of step 13166 have datatype=18 WARNING : type of input 5 of step 13164 doesn't seem to be define in the database( WARNING : output 0 of step 13166 have datatype=11 whereas input 5 of step 13164 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 1 of step 13158 have datatype=7 whereas input 2 of step 13157 have datatype=None WARNING : type of output 3 of step 13157 doesn't seem to be define in the database( WARNING : type of input 3 of step 13161 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13163 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13162 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21222677 AND mptpi.`type`=4855 To do Qualite : 0.39564799928084676 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P21222678_10-03-2025_16_10_23.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 21222678 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 13156 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 13160 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 13160 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 13157 crop_condition is not consistent : 4 used against 2 in the step definition ! WARNING : number of outputs for step 13157 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 13161 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 13159 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 13159 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 13156 doesn't seem to be define in the database( WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 13161 doesn't seem to be define in the database( WARNING : type of input 3 of step 13159 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13163 have datatype=6 whereas input 2 of step 13157 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13162 have datatype=6 whereas input 2 of step 13157 have datatype=None We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 13161 have datatype=10 whereas input 3 of step 13164 have datatype=6 WARNING : output 0 of step 13161 have datatype=10 whereas input 0 of step 13166 have datatype=18 WARNING : type of input 5 of step 13164 doesn't seem to be define in the database( WARNING : output 0 of step 13166 have datatype=11 whereas input 5 of step 13164 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 1 of step 13158 have datatype=7 whereas input 2 of step 13157 have datatype=None WARNING : type of output 3 of step 13157 doesn't seem to be define in the database( WARNING : type of input 3 of step 13161 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13163 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13162 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=21222678 AND mptpi.`type`=4855 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'04022025': {'nb_upload': 4, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1342523624, 1342523598, 1342523593, 1342523589] Looping around the photos to save general results len do output : 1 /21222676Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4741', None, None, None, None, None, None, None, '2654458') ('4741', '21222676', '1342523624', None, None, None, None, None, '2654458') ('4741', None, None, None, None, None, None, None, '2654458') ('4741', '21222676', '1342523598', None, None, None, None, None, '2654458') ('4741', None, None, None, None, None, None, None, '2654458') ('4741', '21222676', '1342523593', None, None, None, None, None, '2654458') ('4741', None, None, None, None, None, None, None, '2654458') ('4741', '21222676', '1342523589', None, None, None, None, None, '2654458') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 5 time used for this insertion : 0.07987833023071289 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.551896333694458 time spend to save output : 0.08072304725646973 total time spend for step 11 : 1.6326193809509277 caffe_path_current : About to save ! 2 After save, about to update current ! ret : 1 len(input) + len(total_photo_id_missing) : 4 set_done_treatment 527.20user 304.12system 16:06.01elapsed 86%CPU (0avgtext+0avgdata 6946024maxresident)k 2696816inputs+590352outputs (45792major+40561671minor)pagefaults 0swaps