python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' -a 4741 -l 30 ' -s datou_current_4741 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 564124 load datou : 4741 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 13156 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 13160 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 13160 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 13157 crop_condition is not consistent : 4 used against 2 in the step definition ! WARNING : number of outputs for step 13157 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 13161 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 13159 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 13159 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 13156 doesn't seem to be define in the database( WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 13161 doesn't seem to be define in the database( WARNING : type of input 3 of step 13159 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13163 have datatype=6 whereas input 2 of step 13157 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13162 have datatype=6 whereas input 2 of step 13157 have datatype=None We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 13161 have datatype=10 whereas input 3 of step 13164 have datatype=6 WARNING : output 0 of step 13161 have datatype=10 whereas input 0 of step 13166 have datatype=18 WARNING : type of input 5 of step 13164 doesn't seem to be define in the database( WARNING : output 0 of step 13166 have datatype=11 whereas input 5 of step 13164 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 1 of step 13158 have datatype=7 whereas input 2 of step 13157 have datatype=None WARNING : type of output 3 of step 13157 doesn't seem to be define in the database( WARNING : type of input 3 of step 13161 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13163 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13162 doesn't seem to be define in the database( DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5732, 'mask_refus_amiens_050123', 16384, 25088, 'mask_refus_amiens_050123', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 1, 5, 13, 23, 59), datetime.datetime(2023, 1, 5, 13, 23, 59)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5972, 'learn_entrant_syctomXV_111023', 16384, 25088, 'learn_entrant_syctomXV_111023', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 10, 11, 15, 57, 35), datetime.datetime(2023, 10, 11, 15, 57, 35)) load thcls load THCL from format json or kwargs add thcl : 3453 in CacheModelConfig load THCL from format json or kwargs add thcl : 3783 in CacheModelConfig load pdts add pdt : 5732 in CacheModelConfig add pdt : 5972 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 list_input_json: [] Current got : datou_id : 4741, datou_cur_ids : ['2966278'] with mtr_portfolio_ids : ['23377954'] and first list_photo_ids : [] new path : /proc/564124/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 13156 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 13160 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 13160 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 13157 crop_condition is not consistent : 4 used against 2 in the step definition ! WARNING : number of outputs for step 13157 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 13161 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 13159 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 13159 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 13156 doesn't seem to be define in the database( WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 13161 doesn't seem to be define in the database( WARNING : type of input 3 of step 13159 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13163 have datatype=6 whereas input 2 of step 13157 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13162 have datatype=6 whereas input 2 of step 13157 have datatype=None We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 13161 have datatype=10 whereas input 3 of step 13164 have datatype=6 WARNING : output 0 of step 13161 have datatype=10 whereas input 0 of step 13166 have datatype=18 WARNING : type of input 5 of step 13164 doesn't seem to be define in the database( WARNING : output 0 of step 13166 have datatype=11 whereas input 5 of step 13164 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 1 of step 13158 have datatype=7 whereas input 2 of step 13157 have datatype=None WARNING : type of output 3 of step 13157 doesn't seem to be define in the database( WARNING : type of input 3 of step 13161 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13163 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13162 doesn't seem to be define in the database( DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, mask_detect, blur_detection, brightness, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , BFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 3 ; length of list_pids : 3 ; length of list_args : 3 time to download the photos : 0.8628096580505371 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Tue May 27 14:21:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5396 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-05-27 14:21:33.683521: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-27 14:21:33.715157: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-27 14:21:33.717579: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f209c000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-27 14:21:33.717650: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-27 14:21:33.722246: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-27 14:21:33.861509: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3d4c8fb0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-27 14:21:33.861551: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-27 14:21:33.862521: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-27 14:21:33.862888: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-27 14:21:33.865218: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-27 14:21:33.867377: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-27 14:21:33.867760: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-27 14:21:33.871204: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-27 14:21:33.872653: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-27 14:21:33.880899: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-27 14:21:33.882548: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-27 14:21:33.882720: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-27 14:21:33.883692: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-27 14:21:33.883719: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-27 14:21:33.883734: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-27 14:21:33.885341: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4930 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-05-27 14:21:34.210836: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-27 14:21:34.210969: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-27 14:21:34.210998: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-27 14:21:34.211022: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-27 14:21:34.211044: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-27 14:21:34.211065: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-27 14:21:34.211086: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-27 14:21:34.211109: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-27 14:21:34.212384: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-27 14:21:34.213830: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-27 14:21:34.213889: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-27 14:21:34.213915: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-27 14:21:34.213939: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-27 14:21:34.213973: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-27 14:21:34.213998: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-27 14:21:34.214024: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-27 14:21:34.214048: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-27 14:21:34.215361: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-27 14:21:34.215405: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-27 14:21:34.215417: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-27 14:21:34.215429: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-27 14:21:34.216773: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4930 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl3783 thcls : [{'id': 3783, 'mtr_user_id': 31, 'name': 'learn_entrant_syctomXV_111023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'photo_desc_type': 5972, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 3783, 'mtr_user_id': 31, 'name': 'learn_entrant_syctomXV_111023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'photo_desc_type': 5972, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5972 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5972, 'learn_entrant_syctomXV_111023', 16384, 25088, 'learn_entrant_syctomXV_111023', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 10, 11, 15, 57, 35), datetime.datetime(2023, 10, 11, 15, 57, 35)) {'thcl': {'id': 3783, 'mtr_user_id': 31, 'name': 'learn_entrant_syctomXV_111023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'photo_desc_type': 5972, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'sac', 'textiles', 'verre', 'organique', 'dasri', 'masque', 'encombrant', 'autre_emballage', 'autre_non_emballage', 'environnement'], 'list_hashtags_csv': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'svm_hashtag_type_desc': 5972, 'photo_desc_type': 5972, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'sac', 'textiles', 'verre', 'organique', 'dasri', 'masque', 'encombrant', 'autre_emballage', 'autre_non_emballage', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_entrant_syctomXV_111023 NUM_CLASSES 11 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_entrant_syctomXV_111023 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-05-27 14:21:44.530577: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-27 14:21:44.701765: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_entrant_syctomXV_111023 /data/models_weight/learn_entrant_syctomXV_111023/mask_model.h5 size_local : 256052544 size in s3 : 256052544 create time local : 2023-10-26 16:36:43 create time in s3 : 2023-10-11 13:57:17 mask_model.h5 already exist and didn't need to update list_images length : 3 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -121.82891 max: 151.10000 image_metas shape: (1, 19) min: 0.00000 max: 3120.00000 nb d'objets trouves : 15 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 19) min: 0.00000 max: 3120.00000 nb d'objets trouves : 18 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 9.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -112.39922 max: 151.10000 image_metas shape: (1, 19) min: 0.00000 max: 3120.00000 nb d'objets trouves : 0 Detection mask done ! Trying to reset tf kernel 564515 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 107 tf kernel not reseted sub process len(results) : 3 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 3 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5396 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl3783 Catched exception ! Connect or reconnect ! thcls : [{'id': 3783, 'mtr_user_id': 31, 'name': 'learn_entrant_syctomXV_111023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'photo_desc_type': 5972, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 3783, 'mtr_user_id': 31, 'name': 'learn_entrant_syctomXV_111023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'photo_desc_type': 5972, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5972 ['background', 'sac', 'textiles', 'verre', 'organique', 'dasri', 'masque', 'encombrant', 'autre_emballage', 'autre_non_emballage', 'environnement'] time for calcul the mask position with numpy : 0.047479867935180664 nb_pixel_total : 907163 time to create 1 rle with new method : 0.09638452529907227 length of segment : 669 time for calcul the mask position with numpy : 0.000827789306640625 nb_pixel_total : 23307 time to create 1 rle with old method : 0.025266170501708984 length of segment : 181 time for calcul the mask position with numpy : 0.0007264614105224609 nb_pixel_total : 20451 time to create 1 rle with old method : 0.022465229034423828 length of segment : 191 time for calcul the mask position with numpy : 0.0006556510925292969 nb_pixel_total : 17095 time to create 1 rle with old method : 0.018672943115234375 length of segment : 185 time for calcul the mask position with numpy : 0.0008432865142822266 nb_pixel_total : 16038 time to create 1 rle with old method : 0.01764822006225586 length of segment : 174 time for calcul the mask position with numpy : 0.0007107257843017578 nb_pixel_total : 17264 time to create 1 rle with old method : 0.018944501876831055 length of segment : 164 time for calcul the mask position with numpy : 0.0014469623565673828 nb_pixel_total : 17023 time to create 1 rle with old method : 0.01840686798095703 length of segment : 288 time for calcul the mask position with numpy : 0.0002524852752685547 nb_pixel_total : 7496 time to create 1 rle with old method : 0.008232831954956055 length of segment : 89 time for calcul the mask position with numpy : 0.004628896713256836 nb_pixel_total : 245414 time to create 1 rle with new method : 0.012777566909790039 length of segment : 472 time for calcul the mask position with numpy : 0.0062100887298583984 nb_pixel_total : 191946 time to create 1 rle with new method : 0.011356592178344727 length of segment : 422 time for calcul the mask position with numpy : 0.000453948974609375 nb_pixel_total : 17077 time to create 1 rle with old method : 0.020068645477294922 length of segment : 93 time for calcul the mask position with numpy : 0.0003674030303955078 nb_pixel_total : 9896 time to create 1 rle with old method : 0.011612176895141602 length of segment : 145 time for calcul the mask position with numpy : 0.00027108192443847656 nb_pixel_total : 6792 time to create 1 rle with old method : 0.008151531219482422 length of segment : 111 time for calcul the mask position with numpy : 0.0005040168762207031 nb_pixel_total : 11597 time to create 1 rle with old method : 0.014348030090332031 length of segment : 102 time for calcul the mask position with numpy : 0.0002689361572265625 nb_pixel_total : 9725 time to create 1 rle with old method : 0.011255025863647461 length of segment : 138 time for calcul the mask position with numpy : 0.0012843608856201172 nb_pixel_total : 28302 time to create 1 rle with old method : 0.03271603584289551 length of segment : 209 time for calcul the mask position with numpy : 0.0005786418914794922 nb_pixel_total : 15118 time to create 1 rle with old method : 0.01791071891784668 length of segment : 125 time for calcul the mask position with numpy : 0.004910707473754883 nb_pixel_total : 135709 time to create 1 rle with old method : 0.15828323364257812 length of segment : 311 time for calcul the mask position with numpy : 0.0009791851043701172 nb_pixel_total : 18994 time to create 1 rle with old method : 0.03572726249694824 length of segment : 235 time for calcul the mask position with numpy : 0.0005772113800048828 nb_pixel_total : 10799 time to create 1 rle with old method : 0.012773513793945312 length of segment : 133 time for calcul the mask position with numpy : 0.0014150142669677734 nb_pixel_total : 48593 time to create 1 rle with old method : 0.055613040924072266 length of segment : 202 time for calcul the mask position with numpy : 0.017979145050048828 nb_pixel_total : 789991 time to create 1 rle with new method : 0.22205901145935059 length of segment : 590 time for calcul the mask position with numpy : 0.003000974655151367 nb_pixel_total : 128114 time to create 1 rle with old method : 0.14436101913452148 length of segment : 318 time for calcul the mask position with numpy : 0.0007643699645996094 nb_pixel_total : 16935 time to create 1 rle with old method : 0.019400835037231445 length of segment : 196 time for calcul the mask position with numpy : 0.0007760524749755859 nb_pixel_total : 20664 time to create 1 rle with old method : 0.02365422248840332 length of segment : 157 time for calcul the mask position with numpy : 0.0003762245178222656 nb_pixel_total : 5842 time to create 1 rle with old method : 0.007269620895385742 length of segment : 71 time for calcul the mask position with numpy : 0.0003018379211425781 nb_pixel_total : 6916 time to create 1 rle with old method : 0.008149862289428711 length of segment : 78 time for calcul the mask position with numpy : 0.0006172657012939453 nb_pixel_total : 19653 time to create 1 rle with old method : 0.023400545120239258 length of segment : 198 time for calcul the mask position with numpy : 0.01404881477355957 nb_pixel_total : 462917 time to create 1 rle with new method : 0.046499013900756836 length of segment : 579 time for calcul the mask position with numpy : 0.0006554126739501953 nb_pixel_total : 25593 time to create 1 rle with old method : 0.029628515243530273 length of segment : 161 time for calcul the mask position with numpy : 0.0003445148468017578 nb_pixel_total : 12273 time to create 1 rle with old method : 0.014306306838989258 length of segment : 96 time for calcul the mask position with numpy : 0.00031256675720214844 nb_pixel_total : 7832 time to create 1 rle with old method : 0.009019136428833008 length of segment : 88 time for calcul the mask position with numpy : 0.00021910667419433594 nb_pixel_total : 13504 time to create 1 rle with old method : 0.01574563980102539 length of segment : 143 time spent for convertir_results : 3.194760322570801 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 33 chid ids of type : 4854 Number RLEs to save : 0 save missing photos in datou_result : time spend for datou_step_exec : 31.506455421447754 time spend to save output : 0.08133673667907715 total time spend for step 1 : 31.58779215812683 step2:mask_detect Tue May 27 14:22:02 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5396 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-05-27 14:22:05.150857: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-27 14:22:05.179200: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-27 14:22:05.181166: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f20a0000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-27 14:22:05.181226: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-27 14:22:05.185155: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-27 14:22:05.336365: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3dfd3990 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-27 14:22:05.336442: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-27 14:22:05.337781: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-27 14:22:05.338394: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-27 14:22:05.343946: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-27 14:22:05.347503: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-27 14:22:05.348454: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-27 14:22:05.351119: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-27 14:22:05.352334: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-27 14:22:05.357634: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-27 14:22:05.359009: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-27 14:22:05.359119: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-27 14:22:05.359798: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-27 14:22:05.359817: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-27 14:22:05.359826: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-27 14:22:05.360876: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4928 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-05-27 14:22:05.725642: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-27 14:22:05.725843: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-27 14:22:05.725867: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-27 14:22:05.725886: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-27 14:22:05.725902: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-27 14:22:05.725919: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-27 14:22:05.725936: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-27 14:22:05.725954: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-27 14:22:05.726957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-27 14:22:05.728317: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-27 14:22:05.728381: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-27 14:22:05.728399: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-27 14:22:05.728415: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-27 14:22:05.728431: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-27 14:22:05.728447: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-27 14:22:05.728462: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-27 14:22:05.728479: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-27 14:22:05.729434: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-27 14:22:05.729483: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-27 14:22:05.729492: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-27 14:22:05.729500: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-27 14:22:05.730523: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4928 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl3453 thcls : [{'id': 3453, 'mtr_user_id': 31, 'name': 'mask_refus_amiens_050123', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4485, 'photo_desc_type': 5732, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}] thcl {'id': 3453, 'mtr_user_id': 31, 'name': 'mask_refus_amiens_050123', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4485, 'photo_desc_type': 5732, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5732 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5732, 'mask_refus_amiens_050123', 16384, 25088, 'mask_refus_amiens_050123', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 1, 5, 13, 23, 59), datetime.datetime(2023, 1, 5, 13, 23, 59)) {'thcl': {'id': 3453, 'mtr_user_id': 31, 'name': 'mask_refus_amiens_050123', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4485, 'photo_desc_type': 5732, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'], 'list_hashtags_csv': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4485, 'svm_hashtag_type_desc': 5732, 'photo_desc_type': 5732, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_refus_amiens_050123 NUM_CLASSES 9 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : mask_refus_amiens_050123 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-05-27 14:22:16.679009: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-27 14:22:17.049714: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-27 14:22:18.736436: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.737007: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.60G (3865470464 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.737479: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 3.24G (3478923264 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.737945: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.92G (3131030784 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.738411: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.62G (2817927680 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.738874: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.36G (2536134912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.739389: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 2.12G (2282521344 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.739419: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-27 14:22:18.739937: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.739953: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-27 14:22:18.746301: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.746376: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-27 14:22:18.746863: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.746880: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-27 14:22:18.752842: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.752882: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 466.56MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-27 14:22:18.753416: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.753433: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 466.56MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-27 14:22:18.779709: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.779786: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-27 14:22:18.780309: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.780325: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-27 14:22:18.785689: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.785726: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 243.25MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-27 14:22:18.786205: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.786220: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 243.25MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-05-27 14:22:18.814729: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.815267: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.816858: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.817367: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.853211: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.853751: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.855635: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.856166: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.881214: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.881766: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.883402: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.883933: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.889333: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.889823: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.891425: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.891949: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.897516: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.898030: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.899508: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.900028: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.926400: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.926932: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.927447: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.927971: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.931467: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.932000: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.947912: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.948461: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.948954: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.949456: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.961915: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.962420: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.962947: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.963446: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.967662: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.968181: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.972725: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.973277: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.985490: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.985983: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.990117: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.990613: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.991099: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.991607: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.992294: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:18.992773: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.003585: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.004114: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.004621: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.005094: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.005567: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.006038: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.006508: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.006996: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.016223: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.016736: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.022799: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.023299: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.069793: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.070291: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.070778: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.071294: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.079217: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.079731: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.096789: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.097457: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.097952: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.098426: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.102434: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.102944: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.103455: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.103983: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.104980: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.104997: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-05-27 14:22:19.115320: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.115842: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.125047: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.125545: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.126047: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.126526: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.127043: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-05-27 14:22:19.127558: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/mask_refus_amiens_050123 /data/models_weight/mask_refus_amiens_050123/mask_model.h5 size_local : 256009536 size in s3 : 256009536 create time local : 2023-01-05 13:44:07 create time in s3 : 2023-01-05 12:23:52 mask_model.h5 already exist and didn't need to update list_images length : 3 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -121.82891 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 3120.00000 nb d'objets trouves : 59 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 3120.00000 nb d'objets trouves : 37 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 9.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -112.39922 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 3120.00000 nb d'objets trouves : 2 Detection mask done ! Trying to reset tf kernel 566230 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 645 tf kernel not reseted sub process len(results) : 3 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 3 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 1838 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl3453 Catched exception ! Connect or reconnect ! thcls : [{'id': 3453, 'mtr_user_id': 31, 'name': 'mask_refus_amiens_050123', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4485, 'photo_desc_type': 5732, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}] thcl {'id': 3453, 'mtr_user_id': 31, 'name': 'mask_refus_amiens_050123', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4485, 'photo_desc_type': 5732, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5732 ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'] time for calcul the mask position with numpy : 0.0021333694458007812 nb_pixel_total : 45826 time to create 1 rle with old method : 0.054926395416259766 length of segment : 287 time for calcul the mask position with numpy : 0.0007886886596679688 nb_pixel_total : 36079 time to create 1 rle with old method : 0.04619622230529785 length of segment : 199 time for calcul the mask position with numpy : 0.10161066055297852 nb_pixel_total : 841686 time to create 1 rle with new method : 0.11327481269836426 length of segment : 987 time for calcul the mask position with numpy : 0.002805471420288086 nb_pixel_total : 74422 time to create 1 rle with old method : 0.09672999382019043 length of segment : 266 time for calcul the mask position with numpy : 0.0007193088531494141 nb_pixel_total : 32138 time to create 1 rle with old method : 0.03709220886230469 length of segment : 234 time for calcul the mask position with numpy : 0.0035321712493896484 nb_pixel_total : 190028 time to create 1 rle with new method : 0.007912397384643555 length of segment : 569 time for calcul the mask position with numpy : 0.006826639175415039 nb_pixel_total : 315105 time to create 1 rle with new method : 0.01610255241394043 length of segment : 572 time for calcul the mask position with numpy : 0.0008015632629394531 nb_pixel_total : 16089 time to create 1 rle with old method : 0.018889665603637695 length of segment : 141 time for calcul the mask position with numpy : 0.0013527870178222656 nb_pixel_total : 28188 time to create 1 rle with old method : 0.04240298271179199 length of segment : 196 time for calcul the mask position with numpy : 0.0007486343383789062 nb_pixel_total : 15282 time to create 1 rle with old method : 0.018039703369140625 length of segment : 187 time for calcul the mask position with numpy : 0.0005118846893310547 nb_pixel_total : 8414 time to create 1 rle with old method : 0.010290145874023438 length of segment : 101 time for calcul the mask position with numpy : 0.00197601318359375 nb_pixel_total : 68586 time to create 1 rle with old method : 0.0807182788848877 length of segment : 259 time for calcul the mask position with numpy : 0.005096912384033203 nb_pixel_total : 157490 time to create 1 rle with new method : 0.013204097747802734 length of segment : 377 time for calcul the mask position with numpy : 0.006109714508056641 nb_pixel_total : 312309 time to create 1 rle with new method : 0.015080928802490234 length of segment : 720 time for calcul the mask position with numpy : 0.001348257064819336 nb_pixel_total : 31553 time to create 1 rle with old method : 0.054268836975097656 length of segment : 214 time for calcul the mask position with numpy : 0.0009989738464355469 nb_pixel_total : 16521 time to create 1 rle with old method : 0.02016139030456543 length of segment : 197 time for calcul the mask position with numpy : 0.001135110855102539 nb_pixel_total : 39909 time to create 1 rle with old method : 0.051851511001586914 length of segment : 253 time for calcul the mask position with numpy : 0.00058746337890625 nb_pixel_total : 25724 time to create 1 rle with old method : 0.03562760353088379 length of segment : 233 time for calcul the mask position with numpy : 0.0012412071228027344 nb_pixel_total : 26891 time to create 1 rle with old method : 0.0514378547668457 length of segment : 209 time for calcul the mask position with numpy : 0.0006849765777587891 nb_pixel_total : 20758 time to create 1 rle with old method : 0.024892330169677734 length of segment : 191 time for calcul the mask position with numpy : 0.0014717578887939453 nb_pixel_total : 46345 time to create 1 rle with old method : 0.05453634262084961 length of segment : 341 time for calcul the mask position with numpy : 0.004991769790649414 nb_pixel_total : 257646 time to create 1 rle with new method : 0.024164915084838867 length of segment : 481 time for calcul the mask position with numpy : 0.0025255680084228516 nb_pixel_total : 73951 time to create 1 rle with old method : 0.08775782585144043 length of segment : 360 time for calcul the mask position with numpy : 0.0026276111602783203 nb_pixel_total : 95919 time to create 1 rle with old method : 0.12590980529785156 length of segment : 235 time for calcul the mask position with numpy : 0.0004661083221435547 nb_pixel_total : 9866 time to create 1 rle with old method : 0.012242555618286133 length of segment : 164 time for calcul the mask position with numpy : 0.0002601146697998047 nb_pixel_total : 8008 time to create 1 rle with old method : 0.009325504302978516 length of segment : 127 time for calcul the mask position with numpy : 0.0021185874938964844 nb_pixel_total : 106262 time to create 1 rle with old method : 0.1227712631225586 length of segment : 362 time for calcul the mask position with numpy : 0.0015370845794677734 nb_pixel_total : 62455 time to create 1 rle with old method : 0.07107400894165039 length of segment : 268 time for calcul the mask position with numpy : 0.002068042755126953 nb_pixel_total : 114205 time to create 1 rle with old method : 0.12580513954162598 length of segment : 407 time for calcul the mask position with numpy : 0.0008451938629150391 nb_pixel_total : 39829 time to create 1 rle with old method : 0.05390310287475586 length of segment : 351 time for calcul the mask position with numpy : 0.0005805492401123047 nb_pixel_total : 19990 time to create 1 rle with old method : 0.026676654815673828 length of segment : 178 time for calcul the mask position with numpy : 0.004507541656494141 nb_pixel_total : 150828 time to create 1 rle with new method : 0.012398481369018555 length of segment : 655 time for calcul the mask position with numpy : 0.0009772777557373047 nb_pixel_total : 36380 time to create 1 rle with old method : 0.04107332229614258 length of segment : 205 time for calcul the mask position with numpy : 0.00566554069519043 nb_pixel_total : 225826 time to create 1 rle with new method : 0.017313480377197266 length of segment : 584 time for calcul the mask position with numpy : 0.0009417533874511719 nb_pixel_total : 46710 time to create 1 rle with old method : 0.05744576454162598 length of segment : 224 time for calcul the mask position with numpy : 0.0009386539459228516 nb_pixel_total : 30020 time to create 1 rle with old method : 0.03762555122375488 length of segment : 193 time for calcul the mask position with numpy : 0.001825571060180664 nb_pixel_total : 65187 time to create 1 rle with old method : 0.07705140113830566 length of segment : 275 time for calcul the mask position with numpy : 0.0025691986083984375 nb_pixel_total : 104895 time to create 1 rle with old method : 0.12058544158935547 length of segment : 360 time for calcul the mask position with numpy : 0.0012664794921875 nb_pixel_total : 41054 time to create 1 rle with old method : 0.046039581298828125 length of segment : 427 time for calcul the mask position with numpy : 0.0006043910980224609 nb_pixel_total : 21027 time to create 1 rle with old method : 0.02484583854675293 length of segment : 193 time for calcul the mask position with numpy : 0.000518798828125 nb_pixel_total : 24833 time to create 1 rle with old method : 0.029906511306762695 length of segment : 115 time for calcul the mask position with numpy : 0.0036237239837646484 nb_pixel_total : 136093 time to create 1 rle with old method : 0.2150731086730957 length of segment : 334 time for calcul the mask position with numpy : 0.0013279914855957031 nb_pixel_total : 59813 time to create 1 rle with old method : 0.07415604591369629 length of segment : 235 time for calcul the mask position with numpy : 0.0010988712310791016 nb_pixel_total : 38872 time to create 1 rle with old method : 0.044240713119506836 length of segment : 306 time for calcul the mask position with numpy : 0.002655029296875 nb_pixel_total : 74616 time to create 1 rle with old method : 0.08628034591674805 length of segment : 314 time for calcul the mask position with numpy : 0.0178680419921875 nb_pixel_total : 406119 time to create 1 rle with new method : 0.29980015754699707 length of segment : 807 time for calcul the mask position with numpy : 0.0010211467742919922 nb_pixel_total : 38637 time to create 1 rle with old method : 0.042815208435058594 length of segment : 264 time for calcul the mask position with numpy : 0.0010368824005126953 nb_pixel_total : 36349 time to create 1 rle with old method : 0.041570186614990234 length of segment : 164 time for calcul the mask position with numpy : 0.002610921859741211 nb_pixel_total : 107173 time to create 1 rle with old method : 0.11957550048828125 length of segment : 745 time for calcul the mask position with numpy : 0.0024819374084472656 nb_pixel_total : 121401 time to create 1 rle with old method : 0.13286805152893066 length of segment : 541 time for calcul the mask position with numpy : 0.015455484390258789 nb_pixel_total : 377902 time to create 1 rle with new method : 0.13363885879516602 length of segment : 1415 time for calcul the mask position with numpy : 0.0019576549530029297 nb_pixel_total : 98865 time to create 1 rle with old method : 0.11095905303955078 length of segment : 297 time for calcul the mask position with numpy : 0.000888824462890625 nb_pixel_total : 30774 time to create 1 rle with old method : 0.03536725044250488 length of segment : 207 time for calcul the mask position with numpy : 0.020720243453979492 nb_pixel_total : 862987 time to create 1 rle with new method : 1.1615393161773682 length of segment : 922 time for calcul the mask position with numpy : 0.0013277530670166016 nb_pixel_total : 72522 time to create 1 rle with old method : 0.07984185218811035 length of segment : 455 time for calcul the mask position with numpy : 0.0044841766357421875 nb_pixel_total : 130944 time to create 1 rle with old method : 0.14558696746826172 length of segment : 672 time for calcul the mask position with numpy : 0.0005776882171630859 nb_pixel_total : 25674 time to create 1 rle with old method : 0.029061317443847656 length of segment : 193 time for calcul the mask position with numpy : 0.0009517669677734375 nb_pixel_total : 41515 time to create 1 rle with old method : 0.04739975929260254 length of segment : 219 time for calcul the mask position with numpy : 0.0009031295776367188 nb_pixel_total : 42553 time to create 1 rle with old method : 0.04860043525695801 length of segment : 523 time for calcul the mask position with numpy : 0.007887601852416992 nb_pixel_total : 486386 time to create 1 rle with new method : 0.03042435646057129 length of segment : 594 time for calcul the mask position with numpy : 0.0012850761413574219 nb_pixel_total : 68370 time to create 1 rle with old method : 0.08572220802307129 length of segment : 266 time for calcul the mask position with numpy : 0.008255720138549805 nb_pixel_total : 389150 time to create 1 rle with new method : 0.8320589065551758 length of segment : 1156 time for calcul the mask position with numpy : 0.0018854141235351562 nb_pixel_total : 120868 time to create 1 rle with old method : 0.13547420501708984 length of segment : 224 time for calcul the mask position with numpy : 0.023520469665527344 nb_pixel_total : 797244 time to create 1 rle with new method : 0.09563231468200684 length of segment : 904 time for calcul the mask position with numpy : 0.01016998291015625 nb_pixel_total : 633976 time to create 1 rle with new method : 0.06626605987548828 length of segment : 971 time for calcul the mask position with numpy : 0.0008728504180908203 nb_pixel_total : 35670 time to create 1 rle with old method : 0.04619550704956055 length of segment : 268 time for calcul the mask position with numpy : 0.0009284019470214844 nb_pixel_total : 47104 time to create 1 rle with old method : 0.05453920364379883 length of segment : 242 time for calcul the mask position with numpy : 0.0007359981536865234 nb_pixel_total : 33046 time to create 1 rle with old method : 0.037758588790893555 length of segment : 235 time for calcul the mask position with numpy : 0.006165266036987305 nb_pixel_total : 337755 time to create 1 rle with new method : 0.024429798126220703 length of segment : 599 time for calcul the mask position with numpy : 0.0008156299591064453 nb_pixel_total : 39029 time to create 1 rle with old method : 0.05095171928405762 length of segment : 198 time for calcul the mask position with numpy : 0.0003790855407714844 nb_pixel_total : 10001 time to create 1 rle with old method : 0.012923717498779297 length of segment : 89 time for calcul the mask position with numpy : 0.006844520568847656 nb_pixel_total : 302025 time to create 1 rle with new method : 0.016927242279052734 length of segment : 676 time for calcul the mask position with numpy : 0.0007755756378173828 nb_pixel_total : 25107 time to create 1 rle with old method : 0.03134870529174805 length of segment : 241 time for calcul the mask position with numpy : 0.002709627151489258 nb_pixel_total : 156797 time to create 1 rle with new method : 0.007812738418579102 length of segment : 364 time for calcul the mask position with numpy : 0.002251863479614258 nb_pixel_total : 124736 time to create 1 rle with old method : 0.13888192176818848 length of segment : 802 time for calcul the mask position with numpy : 0.0008518695831298828 nb_pixel_total : 41939 time to create 1 rle with old method : 0.04691743850708008 length of segment : 195 time for calcul the mask position with numpy : 0.001241922378540039 nb_pixel_total : 59558 time to create 1 rle with old method : 0.06626534461975098 length of segment : 603 time for calcul the mask position with numpy : 0.0012526512145996094 nb_pixel_total : 61442 time to create 1 rle with old method : 0.0804746150970459 length of segment : 266 time for calcul the mask position with numpy : 0.0011560916900634766 nb_pixel_total : 70243 time to create 1 rle with old method : 0.0807962417602539 length of segment : 219 time for calcul the mask position with numpy : 0.00044655799865722656 nb_pixel_total : 21529 time to create 1 rle with old method : 0.02535390853881836 length of segment : 157 time for calcul the mask position with numpy : 0.0008082389831542969 nb_pixel_total : 33303 time to create 1 rle with old method : 0.03808426856994629 length of segment : 194 time for calcul the mask position with numpy : 0.007941007614135742 nb_pixel_total : 240709 time to create 1 rle with new method : 0.016335010528564453 length of segment : 533 time for calcul the mask position with numpy : 0.0034775733947753906 nb_pixel_total : 168373 time to create 1 rle with new method : 0.009365320205688477 length of segment : 405 time for calcul the mask position with numpy : 0.0018324851989746094 nb_pixel_total : 112998 time to create 1 rle with old method : 0.12737727165222168 length of segment : 356 time for calcul the mask position with numpy : 0.0016202926635742188 nb_pixel_total : 99074 time to create 1 rle with old method : 0.11161684989929199 length of segment : 372 time for calcul the mask position with numpy : 0.0001983642578125 nb_pixel_total : 8981 time to create 1 rle with old method : 0.010086297988891602 length of segment : 73 time for calcul the mask position with numpy : 0.0010290145874023438 nb_pixel_total : 45776 time to create 1 rle with old method : 0.051517486572265625 length of segment : 275 time for calcul the mask position with numpy : 0.001348733901977539 nb_pixel_total : 68726 time to create 1 rle with old method : 0.08656120300292969 length of segment : 254 time for calcul the mask position with numpy : 0.002221822738647461 nb_pixel_total : 141528 time to create 1 rle with old method : 0.17332720756530762 length of segment : 359 time for calcul the mask position with numpy : 0.0008604526519775391 nb_pixel_total : 44221 time to create 1 rle with old method : 0.0525822639465332 length of segment : 244 time for calcul the mask position with numpy : 0.0010867118835449219 nb_pixel_total : 73491 time to create 1 rle with old method : 0.08478665351867676 length of segment : 259 time for calcul the mask position with numpy : 0.011276006698608398 nb_pixel_total : 519232 time to create 1 rle with new method : 1.00691556930542 length of segment : 825 time for calcul the mask position with numpy : 0.0013113021850585938 nb_pixel_total : 81231 time to create 1 rle with old method : 0.09315872192382812 length of segment : 268 time for calcul the mask position with numpy : 0.02996969223022461 nb_pixel_total : 801042 time to create 1 rle with new method : 0.7194490432739258 length of segment : 1075 time for calcul the mask position with numpy : 0.0004343986511230469 nb_pixel_total : 21515 time to create 1 rle with old method : 0.024210214614868164 length of segment : 137 time for calcul the mask position with numpy : 0.00030303001403808594 nb_pixel_total : 19534 time to create 1 rle with old method : 0.031056642532348633 length of segment : 128 time for calcul the mask position with numpy : 0.9301047325134277 nb_pixel_total : 8253641 time to create 1 rle with new method : 0.9769711494445801 length of segment : 2902 time for calcul the mask position with numpy : 0.00042939186096191406 nb_pixel_total : 20702 time to create 1 rle with old method : 0.023237228393554688 length of segment : 197 time spent for convertir_results : 20.79567790031433 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 131 chid ids of type : 4854 +++++++++++++++++++++++++++++++++++Number RLEs to save : 0 save missing photos in datou_result : time spend for datou_step_exec : 64.812255859375 time spend to save output : 0.12222886085510254 total time spend for step 2 : 64.9344847202301 step3:blur_detection Tue May 27 14:23:07 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure inside step blur_detection methode: variance laplacian (3120, 3120) 3180.1069132243583 (3120, 3120) 1384.9586911260333 (3120, 3120) 121.29620009356601 Inside saveOutput : final : False verbose : 0 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 3 time used for this insertion : 0.013308048248291016 begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 3 time used for this insertion : 0.011460304260253906 save missing photos in datou_result : time spend for datou_step_exec : 1.0813038349151611 time spend to save output : 0.02974677085876465 total time spend for step 3 : 1.1110506057739258 step4:brightness Tue May 27 14:23:08 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure inside step calcul brightness treat image : temp/1748348489_564124_1361298409_cf80260ec1503be2f990f0c8e94209f6.jpg treat image : temp/1748348489_564124_1361298402_dc366ba342d1eb0ef76bb1631e9c953d.jpg treat image : temp/1748348489_564124_1361298257_410c3104eb616ef449f2db3a3cf191ea.jpg Inside saveOutput : final : False verbose : 0 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 3 time used for this insertion : 0.015538454055786133 begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 3 time used for this insertion : 0.014993667602539062 save missing photos in datou_result : time spend for datou_step_exec : 3.483062982559204 time spend to save output : 0.035199642181396484 total time spend for step 4 : 3.5182626247406006 step5:rle_unique_nms_with_priority Tue May 27 14:23:11 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms batch 1 Loaded 131 chid ids of type : 4854 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++nb_obj : 74 nb_hashtags : 8 time to prepare the origin masks : 44.70303559303284 time for calcul the mask position with numpy : 0.3963782787322998 nb_pixel_total : 3824255 time to create 1 rle with new method : 0.9369227886199951 time for calcul the mask position with numpy : 0.0493621826171875 nb_pixel_total : 906120 time to create 1 rle with new method : 0.7883079051971436 time for calcul the mask position with numpy : 0.042542219161987305 nb_pixel_total : 50035 time to create 1 rle with old method : 0.05573391914367676 time for calcul the mask position with numpy : 0.04013514518737793 nb_pixel_total : 48387 time to create 1 rle with old method : 0.0532381534576416 time for calcul the mask position with numpy : 0.03946375846862793 nb_pixel_total : 32026 time to create 1 rle with old method : 0.035161495208740234 time for calcul the mask position with numpy : 0.04035139083862305 nb_pixel_total : 7118 time to create 1 rle with old method : 0.008038759231567383 time for calcul the mask position with numpy : 0.04025721549987793 nb_pixel_total : 26808 time to create 1 rle with old method : 0.03014659881591797 time for calcul the mask position with numpy : 0.04036116600036621 nb_pixel_total : 16926 time to create 1 rle with old method : 0.019681453704833984 time for calcul the mask position with numpy : 0.04194808006286621 nb_pixel_total : 30032 time to create 1 rle with old method : 0.036916255950927734 time for calcul the mask position with numpy : 0.04024553298950195 nb_pixel_total : 38068 time to create 1 rle with old method : 0.04429030418395996 time for calcul the mask position with numpy : 0.0448458194732666 nb_pixel_total : 20113 time to create 1 rle with old method : 0.025847196578979492 time for calcul the mask position with numpy : 0.04306769371032715 nb_pixel_total : 95464 time to create 1 rle with old method : 0.10861372947692871 time for calcul the mask position with numpy : 0.042163848876953125 nb_pixel_total : 64961 time to create 1 rle with old method : 0.09065747261047363 time for calcul the mask position with numpy : 0.04024219512939453 nb_pixel_total : 28067 time to create 1 rle with old method : 0.03157544136047363 time for calcul the mask position with numpy : 0.040090322494506836 nb_pixel_total : 5 time to create 1 rle with old method : 3.1948089599609375e-05 time for calcul the mask position with numpy : 0.041487932205200195 nb_pixel_total : 331413 time to create 1 rle with new method : 0.8959414958953857 time for calcul the mask position with numpy : 0.03902792930603027 nb_pixel_total : 271 time to create 1 rle with old method : 0.00041484832763671875 time for calcul the mask position with numpy : 0.03856015205383301 nb_pixel_total : 9860 time to create 1 rle with old method : 0.01083993911743164 time for calcul the mask position with numpy : 0.03940606117248535 nb_pixel_total : 157059 time to create 1 rle with new method : 1.1848504543304443 time for calcul the mask position with numpy : 0.03849315643310547 nb_pixel_total : 31367 time to create 1 rle with old method : 0.0338590145111084 time for calcul the mask position with numpy : 0.03845667839050293 nb_pixel_total : 29815 time to create 1 rle with old method : 0.032170772552490234 time for calcul the mask position with numpy : 0.03838706016540527 nb_pixel_total : 15245 time to create 1 rle with old method : 0.016984224319458008 time for calcul the mask position with numpy : 0.04009389877319336 nb_pixel_total : 1309 time to create 1 rle with old method : 0.0016331672668457031 time for calcul the mask position with numpy : 0.038718461990356445 nb_pixel_total : 16039 time to create 1 rle with old method : 0.017510652542114258 time for calcul the mask position with numpy : 0.038758277893066406 nb_pixel_total : 23250 time to create 1 rle with old method : 0.037215471267700195 time for calcul the mask position with numpy : 0.04549837112426758 nb_pixel_total : 43967 time to create 1 rle with old method : 0.04842209815979004 time for calcul the mask position with numpy : 0.03919029235839844 nb_pixel_total : 3968 time to create 1 rle with old method : 0.00445246696472168 time for calcul the mask position with numpy : 0.039911746978759766 nb_pixel_total : 17045 time to create 1 rle with old method : 0.019250154495239258 time for calcul the mask position with numpy : 0.03952765464782715 nb_pixel_total : 150312 time to create 1 rle with new method : 0.8292431831359863 time for calcul the mask position with numpy : 0.04052877426147461 nb_pixel_total : 106070 time to create 1 rle with old method : 0.12300419807434082 time for calcul the mask position with numpy : 0.040899038314819336 nb_pixel_total : 21710 time to create 1 rle with old method : 0.03507232666015625 time for calcul the mask position with numpy : 0.04552960395812988 nb_pixel_total : 145813 time to create 1 rle with old method : 0.16179466247558594 time for calcul the mask position with numpy : 0.04000544548034668 nb_pixel_total : 11163 time to create 1 rle with old method : 0.012613296508789062 time for calcul the mask position with numpy : 0.04007411003112793 nb_pixel_total : 20716 time to create 1 rle with old method : 0.023192882537841797 time for calcul the mask position with numpy : 0.04071784019470215 nb_pixel_total : 73843 time to create 1 rle with old method : 0.08283233642578125 time for calcul the mask position with numpy : 0.04014992713928223 nb_pixel_total : 6207 time to create 1 rle with old method : 0.007154226303100586 time for calcul the mask position with numpy : 0.03986930847167969 nb_pixel_total : 19569 time to create 1 rle with old method : 0.022016525268554688 time for calcul the mask position with numpy : 0.04031229019165039 nb_pixel_total : 17222 time to create 1 rle with old method : 0.019333362579345703 time for calcul the mask position with numpy : 0.03991818428039551 nb_pixel_total : 36273 time to create 1 rle with old method : 0.040400028228759766 time for calcul the mask position with numpy : 0.03954911231994629 nb_pixel_total : 112572 time to create 1 rle with old method : 0.12758827209472656 time for calcul the mask position with numpy : 0.0399785041809082 nb_pixel_total : 45676 time to create 1 rle with old method : 0.04958224296569824 time for calcul the mask position with numpy : 0.03917646408081055 nb_pixel_total : 1074 time to create 1 rle with old method : 0.0015635490417480469 time for calcul the mask position with numpy : 0.03961372375488281 nb_pixel_total : 98299 time to create 1 rle with old method : 0.10674929618835449 time for calcul the mask position with numpy : 0.04109668731689453 nb_pixel_total : 256725 time to create 1 rle with new method : 0.7679293155670166 time for calcul the mask position with numpy : 0.03891277313232422 nb_pixel_total : 7981 time to create 1 rle with old method : 0.008617877960205078 time for calcul the mask position with numpy : 0.03902721405029297 nb_pixel_total : 21 time to create 1 rle with old method : 4.7206878662109375e-05 time for calcul the mask position with numpy : 0.03919863700866699 nb_pixel_total : 38588 time to create 1 rle with old method : 0.041382551193237305 time for calcul the mask position with numpy : 0.03850865364074707 nb_pixel_total : 9797 time to create 1 rle with old method : 0.011670112609863281 time for calcul the mask position with numpy : 0.0403444766998291 nb_pixel_total : 74366 time to create 1 rle with old method : 0.08857178688049316 time for calcul the mask position with numpy : 0.04591536521911621 nb_pixel_total : 314635 time to create 1 rle with new method : 1.0167326927185059 time for calcul the mask position with numpy : 0.04167461395263672 nb_pixel_total : 189844 time to create 1 rle with new method : 0.7946338653564453 time for calcul the mask position with numpy : 0.04081273078918457 nb_pixel_total : 68460 time to create 1 rle with old method : 0.07446885108947754 time for calcul the mask position with numpy : 0.03957247734069824 nb_pixel_total : 39822 time to create 1 rle with old method : 0.04345965385437012 time for calcul the mask position with numpy : 0.04043006896972656 nb_pixel_total : 35960 time to create 1 rle with old method : 0.04038643836975098 time for calcul the mask position with numpy : 0.04082345962524414 nb_pixel_total : 20558 time to create 1 rle with old method : 0.024396419525146484 time for calcul the mask position with numpy : 0.040901899337768555 nb_pixel_total : 19712 time to create 1 rle with old method : 0.022220373153686523 time for calcul the mask position with numpy : 0.04088091850280762 nb_pixel_total : 46191 time to create 1 rle with old method : 0.054259300231933594 time for calcul the mask position with numpy : 0.04115700721740723 nb_pixel_total : 8322 time to create 1 rle with old method : 0.009462356567382812 time for calcul the mask position with numpy : 0.04072833061218262 nb_pixel_total : 113907 time to create 1 rle with old method : 0.12729835510253906 time for calcul the mask position with numpy : 0.03985595703125 nb_pixel_total : 139508 time to create 1 rle with old method : 0.15327167510986328 time for calcul the mask position with numpy : 0.045499563217163086 nb_pixel_total : 857720 time to create 1 rle with new method : 0.8972222805023193 time for calcul the mask position with numpy : 0.03898906707763672 nb_pixel_total : 36156 time to create 1 rle with old method : 0.03867030143737793 time for calcul the mask position with numpy : 0.03896951675415039 nb_pixel_total : 24766 time to create 1 rle with old method : 0.027708053588867188 time for calcul the mask position with numpy : 0.03897500038146973 nb_pixel_total : 8519 time to create 1 rle with old method : 0.00934910774230957 time for calcul the mask position with numpy : 0.039107561111450195 nb_pixel_total : 17042 time to create 1 rle with old method : 0.018744468688964844 time for calcul the mask position with numpy : 0.039886474609375 nb_pixel_total : 74149 time to create 1 rle with old method : 0.08153057098388672 time for calcul the mask position with numpy : 0.040457963943481445 nb_pixel_total : 1067 time to create 1 rle with old method : 0.0013885498046875 time for calcul the mask position with numpy : 0.04253888130187988 nb_pixel_total : 311486 time to create 1 rle with new method : 0.7249064445495605 time for calcul the mask position with numpy : 0.040642738342285156 nb_pixel_total : 121318 time to create 1 rle with old method : 0.1351618766784668 time for calcul the mask position with numpy : 0.0390470027923584 nb_pixel_total : 19000 time to create 1 rle with old method : 0.020561695098876953 time for calcul the mask position with numpy : 0.03876948356628418 nb_pixel_total : 66622 time to create 1 rle with old method : 0.0734093189239502 time for calcul the mask position with numpy : 0.03861880302429199 nb_pixel_total : 3245 time to create 1 rle with old method : 0.0035674571990966797 time for calcul the mask position with numpy : 0.0384976863861084 nb_pixel_total : 51129 time to create 1 rle with old method : 0.06604123115539551 time for calcul the mask position with numpy : 0.0452117919921875 nb_pixel_total : 15860 time to create 1 rle with old method : 0.017604351043701172 time for calcul the mask position with numpy : 0.03860187530517578 nb_pixel_total : 6412 time to create 1 rle with old method : 0.006654977798461914 create new chi : 15.335850954055786 time to delete rle : 0.053337812423706055 batch 1 Loaded 75 chid ids of type : 4855 Number RLEs to save : 41847 TO DO : save crop sub photo not yet done ! save time : 2.4415087699890137 nb_obj : 54 nb_hashtags : 9 time to prepare the origin masks : 29.349385499954224 time for calcul the mask position with numpy : 0.4068150520324707 nb_pixel_total : 3655053 time to create 1 rle with new method : 1.295687198638916 time for calcul the mask position with numpy : 0.0412898063659668 nb_pixel_total : 17106 time to create 1 rle with old method : 0.019099712371826172 time for calcul the mask position with numpy : 0.04211020469665527 nb_pixel_total : 135219 time to create 1 rle with old method : 0.1534111499786377 time for calcul the mask position with numpy : 0.03947854042053223 nb_pixel_total : 8763 time to create 1 rle with old method : 0.009665250778198242 time for calcul the mask position with numpy : 0.04927849769592285 nb_pixel_total : 788537 time to create 1 rle with new method : 1.284637451171875 time for calcul the mask position with numpy : 0.042826175689697266 nb_pixel_total : 5205 time to create 1 rle with old method : 0.0067369937896728516 time for calcul the mask position with numpy : 0.04677605628967285 nb_pixel_total : 336027 time to create 1 rle with new method : 1.0129361152648926 time for calcul the mask position with numpy : 0.041715383529663086 nb_pixel_total : 1196 time to create 1 rle with old method : 0.0016999244689941406 time for calcul the mask position with numpy : 0.041398048400878906 nb_pixel_total : 28728 time to create 1 rle with old method : 0.032758235931396484 time for calcul the mask position with numpy : 0.0414273738861084 nb_pixel_total : 127877 time to create 1 rle with old method : 0.14075374603271484 time for calcul the mask position with numpy : 0.04651069641113281 nb_pixel_total : 755282 time to create 1 rle with new method : 0.9865243434906006 time for calcul the mask position with numpy : 0.046488285064697266 nb_pixel_total : 485316 time to create 1 rle with new method : 0.9520177841186523 time for calcul the mask position with numpy : 0.04290151596069336 nb_pixel_total : 15096 time to create 1 rle with old method : 0.01670384407043457 time for calcul the mask position with numpy : 0.04158782958984375 nb_pixel_total : 12224 time to create 1 rle with old method : 0.013720273971557617 time for calcul the mask position with numpy : 0.04288029670715332 nb_pixel_total : 33177 time to create 1 rle with old method : 0.037435054779052734 time for calcul the mask position with numpy : 0.040357112884521484 nb_pixel_total : 19594 time to create 1 rle with old method : 0.021600961685180664 time for calcul the mask position with numpy : 0.04170942306518555 nb_pixel_total : 45495 time to create 1 rle with old method : 0.05135059356689453 time for calcul the mask position with numpy : 0.04109001159667969 nb_pixel_total : 6880 time to create 1 rle with old method : 0.007750034332275391 time for calcul the mask position with numpy : 0.04141950607299805 nb_pixel_total : 18970 time to create 1 rle with old method : 0.023631572723388672 time for calcul the mask position with numpy : 0.04288077354431152 nb_pixel_total : 238286 time to create 1 rle with new method : 1.2610111236572266 time for calcul the mask position with numpy : 0.04106712341308594 nb_pixel_total : 16908 time to create 1 rle with old method : 0.018882274627685547 time for calcul the mask position with numpy : 0.04050135612487793 nb_pixel_total : 35577 time to create 1 rle with old method : 0.03966808319091797 time for calcul the mask position with numpy : 0.04010319709777832 nb_pixel_total : 80547 time to create 1 rle with old method : 0.08992505073547363 time for calcul the mask position with numpy : 0.040917158126831055 nb_pixel_total : 167521 time to create 1 rle with new method : 1.3889250755310059 time for calcul the mask position with numpy : 0.04039120674133301 nb_pixel_total : 14160 time to create 1 rle with old method : 0.01615762710571289 time for calcul the mask position with numpy : 0.04445767402648926 nb_pixel_total : 1657 time to create 1 rle with old method : 0.0020601749420166016 time for calcul the mask position with numpy : 0.04018092155456543 nb_pixel_total : 28206 time to create 1 rle with old method : 0.03163647651672363 time for calcul the mask position with numpy : 0.04240250587463379 nb_pixel_total : 21428 time to create 1 rle with old method : 0.023845195770263672 time for calcul the mask position with numpy : 0.04026675224304199 nb_pixel_total : 68178 time to create 1 rle with old method : 0.07486414909362793 time for calcul the mask position with numpy : 0.04024505615234375 nb_pixel_total : 44092 time to create 1 rle with old method : 0.04870176315307617 time for calcul the mask position with numpy : 0.040045976638793945 nb_pixel_total : 68 time to create 1 rle with old method : 0.0003256797790527344 time for calcul the mask position with numpy : 0.039710044860839844 nb_pixel_total : 7811 time to create 1 rle with old method : 0.008941411972045898 time for calcul the mask position with numpy : 0.04120063781738281 nb_pixel_total : 288423 time to create 1 rle with new method : 0.9898219108581543 time for calcul the mask position with numpy : 0.03989887237548828 nb_pixel_total : 9847 time to create 1 rle with old method : 0.011343955993652344 time for calcul the mask position with numpy : 0.04262566566467285 nb_pixel_total : 5759 time to create 1 rle with old method : 0.007260322570800781 time for calcul the mask position with numpy : 0.03904080390930176 nb_pixel_total : 7810 time to create 1 rle with old method : 0.008754253387451172 time for calcul the mask position with numpy : 0.039850473403930664 nb_pixel_total : 26233 time to create 1 rle with old method : 0.033172607421875 time for calcul the mask position with numpy : 0.04739642143249512 nb_pixel_total : 388061 time to create 1 rle with new method : 1.0017180442810059 time for calcul the mask position with numpy : 0.040434837341308594 nb_pixel_total : 41688 time to create 1 rle with old method : 0.0475618839263916 time for calcul the mask position with numpy : 0.05090069770812988 nb_pixel_total : 796414 time to create 1 rle with new method : 1.1526846885681152 time for calcul the mask position with numpy : 0.04037666320800781 nb_pixel_total : 38881 time to create 1 rle with old method : 0.04663395881652832 time for calcul the mask position with numpy : 0.043656349182128906 nb_pixel_total : 4346 time to create 1 rle with old method : 0.004879474639892578 time for calcul the mask position with numpy : 0.039392709732055664 nb_pixel_total : 20589 time to create 1 rle with old method : 0.022583723068237305 time for calcul the mask position with numpy : 0.039991140365600586 nb_pixel_total : 46329 time to create 1 rle with old method : 0.05138564109802246 time for calcul the mask position with numpy : 0.04312872886657715 nb_pixel_total : 2913 time to create 1 rle with old method : 0.0047910213470458984 time for calcul the mask position with numpy : 0.051482200622558594 nb_pixel_total : 10756 time to create 1 rle with old method : 0.016794919967651367 time for calcul the mask position with numpy : 0.052454471588134766 nb_pixel_total : 98857 time to create 1 rle with old method : 0.14605283737182617 time for calcul the mask position with numpy : 0.04231572151184082 nb_pixel_total : 70764 time to create 1 rle with old method : 0.07914972305297852 time for calcul the mask position with numpy : 0.044768571853637695 nb_pixel_total : 485714 time to create 1 rle with new method : 0.9786820411682129 time for calcul the mask position with numpy : 0.041066884994506836 nb_pixel_total : 10753 time to create 1 rle with old method : 0.012612581253051758 time for calcul the mask position with numpy : 0.040590763092041016 nb_pixel_total : 8658 time to create 1 rle with old method : 0.010987281799316406 time for calcul the mask position with numpy : 0.04008626937866211 nb_pixel_total : 337 time to create 1 rle with old method : 0.0004315376281738281 time for calcul the mask position with numpy : 0.043239593505859375 nb_pixel_total : 65975 time to create 1 rle with old method : 0.07793998718261719 time for calcul the mask position with numpy : 0.04058647155761719 nb_pixel_total : 77985 time to create 1 rle with old method : 0.10940337181091309 time for calcul the mask position with numpy : 0.05134773254394531 nb_pixel_total : 7124 time to create 1 rle with old method : 0.008202552795410156 create new chi : 16.98620867729187 time to delete rle : 0.004725933074951172 batch 1 Loaded 55 chid ids of type : 4855 Number RLEs to save : 31763 TO DO : save crop sub photo not yet done ! save time : 1.9392457008361816 nb_obj : 1 nb_hashtags : 1 time to prepare the origin masks : 2.7855563163757324 time for calcul the mask position with numpy : 0.08376860618591309 nb_pixel_total : 1482920 time to create 1 rle with new method : 1.1762104034423828 time for calcul the mask position with numpy : 0.7186448574066162 nb_pixel_total : 8251480 time to create 1 rle with new method : 1.0861074924468994 create new chi : 3.1520655155181885 time to delete rle : 0.001039266586303711 batch 1 Loaded 2 chid ids of type : 4855 Number RLEs to save : 8918 TO DO : save crop sub photo not yet done ! save time : 0.5599284172058105 map_output_result : {1361298409: (0.260678595286944, 'Should be the crop_list due to order', 0.3418050313707049), 1361298402: (0.260678595286944, 'Should be the crop_list due to order', 0.44023075449012716), 1361298257: (0.260678595286944, 'Should be the crop_list due to order', 0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1361298409, 1361298402, 1361298257] Looping around the photos to save general results len do output : 3 /1361298409.Didn't retrieve data . /1361298402.Didn't retrieve data . /1361298257.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4741', None, None, None, None, None, None, None, '2966278') ('4741', '23377954', '1361298409', None, None, None, None, None, '2966278') ('4741', None, None, None, None, None, None, None, '2966278') ('4741', '23377954', '1361298402', None, None, None, None, None, '2966278') ('4741', None, None, None, None, None, None, None, '2966278') ('4741', '23377954', '1361298257', None, None, None, None, None, '2966278') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 9 time used for this insertion : 0.012723445892333984 save_final save missing photos in datou_result : time spend for datou_step_exec : 118.09470438957214 time spend to save output : 0.013063430786132812 total time spend for step 5 : 118.10776782035828 step6:crop_condition Tue May 27 14:25:10 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 4855 Loading chi in step crop for list_pids : 3 ! batch 1 Loaded 132 chid ids of type : 4855 begin to crop the class : papier param for this class : {'min_score': 0.1} filtre for class : papier hashtag_id of this class : 492668766 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 18 About to insert : list_path_to_insert length 18 new photo from crops ! About to upload 18 photos upload in portfolio : 3015255 init cache_photo without model_param we have 18 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1748348727_564124 we have uploaded 18 photos in the portfolio 3015255 time of upload the photos Elapsed time : 5.570979595184326 we have finished the crop for the class : papier begin to crop the class : carton param for this class : {'min_score': 0.1} filtre for class : carton hashtag_id of this class : 492774966 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 26 About to insert : list_path_to_insert length 26 new photo from crops ! About to upload 26 photos upload in portfolio : 3015255 init cache_photo without model_param we have 26 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1748348739_564124 we have uploaded 26 photos in the portfolio 3015255 time of upload the photos Elapsed time : 7.995713472366333 we have finished the crop for the class : carton begin to crop the class : metal param for this class : {'min_score': 0.1} filtre for class : metal hashtag_id of this class : 492628673 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 3015255 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1748348749_564124 we have uploaded 6 photos in the portfolio 3015255 time of upload the photos Elapsed time : 1.6407198905944824 we have finished the crop for the class : metal begin to crop the class : pet_clair param for this class : {'min_score': 0.1} filtre for class : pet_clair hashtag_id of this class : 2107755846 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 3015255 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1748348753_564124 we have uploaded 2 photos in the portfolio 3015255 time of upload the photos Elapsed time : 0.8354306221008301 we have finished the crop for the class : pet_clair begin to crop the class : pehd param for this class : {'min_score': 0.1} filtre for class : pehd hashtag_id of this class : 628944319 begin to crop the class : pet_fonce param for this class : {'min_score': 0.1} filtre for class : pet_fonce hashtag_id of this class : 2107755900 begin to crop the class : sac param for this class : {'min_score': 0.1} filtre for class : sac hashtag_id of this class : 492618350 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! About to upload 7 photos upload in portfolio : 3015255 init cache_photo without model_param we have 7 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1748348756_564124 we have uploaded 7 photos in the portfolio 3015255 time of upload the photos Elapsed time : 1.80936861038208 we have finished the crop for the class : sac begin to crop the class : textiles param for this class : {'min_score': 0.1} filtre for class : textiles hashtag_id of this class : 494789423 begin to crop the class : verre param for this class : {'min_score': 0.1} filtre for class : verre hashtag_id of this class : 492631919 begin to crop the class : organique param for this class : {'min_score': 0.1} filtre for class : organique hashtag_id of this class : 530726390 begin to crop the class : dasri param for this class : {'min_score': 0.1} filtre for class : dasri hashtag_id of this class : 1608424396 begin to crop the class : masque param for this class : {'min_score': 0.1} filtre for class : masque hashtag_id of this class : 492626781 begin to crop the class : encombrant param for this class : {'min_score': 0.1} filtre for class : encombrant hashtag_id of this class : 605975549 begin to crop the class : autre param for this class : {'min_score': 0.1} filtre for class : autre hashtag_id of this class : 494826614 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 16 About to insert : list_path_to_insert length 16 new photo from crops ! About to upload 16 photos upload in portfolio : 3015255 init cache_photo without model_param we have 16 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1748348772_564124 we have uploaded 16 photos in the portfolio 3015255 time of upload the photos Elapsed time : 3.7798187732696533 we have finished the crop for the class : autre begin to crop the class : autre_emballage param for this class : {'min_score': 0.1} filtre for class : autre_emballage hashtag_id of this class : 2107760328 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 3015255 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1748348777_564124 we have uploaded 2 photos in the portfolio 3015255 time of upload the photos Elapsed time : 0.6995115280151367 we have finished the crop for the class : autre_emballage begin to crop the class : autre_non_emballage param for this class : {'min_score': 0.1} filtre for class : autre_non_emballage hashtag_id of this class : 2107760329 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! About to upload 9 photos upload in portfolio : 3015255 init cache_photo without model_param we have 9 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1748348779_564124 we have uploaded 9 photos in the portfolio 3015255 time of upload the photos Elapsed time : 3.410912036895752 we have finished the crop for the class : autre_non_emballage begin to crop the class : environnement param for this class : {'min_score': 0.1} filtre for class : environnement hashtag_id of this class : 493012381 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 20 About to insert : list_path_to_insert length 20 new photo from crops ! About to upload 20 photos upload in portfolio : 3015255 init cache_photo without model_param we have 20 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1748348868_564124 we have uploaded 20 photos in the portfolio 3015255 time of upload the photos Elapsed time : 6.009610891342163 we have finished the crop for the class : environnement delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1361298409, 1361298402, 1361298257] Looping around the photos to save general results len do output : 106 /1361300188Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300189Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300190Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300191Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300192Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300193Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300194Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300195Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300196Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300197Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300198Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300199Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300200Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300201Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300203Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300204Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300205Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300206Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300295Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300297Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300299Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300301Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300303Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300305Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300307Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300309Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300311Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300313Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300315Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300317Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300319Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300321Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300323Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300325Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300327Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300329Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300331Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300333Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300335Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300337Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300339Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300341Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300343Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300345Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300394Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300395Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300396Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300397Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300398Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300399Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300404Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300405Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300408Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300409Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300410Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300411Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300412Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300413Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300414Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300483Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300484Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300486Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300487Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300488Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300489Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300491Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300492Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300493Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300495Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300496Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300497Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300498Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300500Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300501Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300502Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300513Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300514Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300530Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300532Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300534Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300536Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300538Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300540Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300542Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300544Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300546Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300778Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300779Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300780Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300781Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300782Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300783Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300784Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300785Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300786Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300787Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300788Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300789Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300790Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300791Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300792Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300793Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300794Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300795Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300796Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1361300797Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4741', None, None, None, None, None, None, None, '2966278') ('4741', '23377954', '1361298409', None, None, None, None, None, '2966278') ('4741', None, None, None, None, None, None, None, '2966278') ('4741', '23377954', '1361298402', None, None, None, None, None, '2966278') ('4741', None, None, None, None, None, None, None, '2966278') ('4741', '23377954', '1361298257', None, None, None, None, None, '2966278') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 321 time used for this insertion : 0.04082942008972168 save_final save missing photos in datou_result : time spend for datou_step_exec : 164.71720266342163 time spend to save output : 0.04344892501831055 total time spend for step 6 : 164.76065158843994 step7:ventilate_hashtags_in_portfolio Tue May 27 14:27:54 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! Iterating over portfolio : 23377954 get user id for portfolio 23377954 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=23377954 AND mptpi.`type`=4855 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','carton','metal','pet_clair','pehd','pet_fonce','pet_opaque','barquette_opaque','film_plastique','ela','sac','textiles','verre','organique','dasri','masque','encombrant','autre_emballage','autre_non_emballage','environnement','autre','flux_dev','mal_croppe','flou','film_dev_souple','sac_om_plein')) AND mptpi.`min_score`=0.1 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=23377954 AND mptpi.`type`=4855 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','carton','metal','pet_clair','pehd','pet_fonce','pet_opaque','barquette_opaque','film_plastique','ela','sac','textiles','verre','organique','dasri','masque','encombrant','autre_emballage','autre_non_emballage','environnement','autre','flux_dev','mal_croppe','flou','film_dev_souple','sac_om_plein')) AND mptpi.`min_score`=0.1 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=23377954 AND mptpi.`type`=4856 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','carton','metal','pet_clair','pehd','pet_fonce','pet_opaque','barquette_opaque','film_plastique','ela','sac','textiles','verre','organique','dasri','masque','encombrant','autre_emballage','autre_non_emballage','environnement','autre','flux_dev','mal_croppe','flou','film_dev_souple','sac_om_plein')) AND mptpi.`min_score`=0.1 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=23377954 AND mptpi.`type`=4856 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','carton','metal','pet_clair','pehd','pet_fonce','pet_opaque','barquette_opaque','film_plastique','ela','sac','textiles','verre','organique','dasri','masque','encombrant','autre_emballage','autre_non_emballage','environnement','autre','flux_dev','mal_croppe','flou','film_dev_souple','sac_om_plein')) AND mptpi.`min_score`=0.1 To do lien utilise dans velours : https://www.fotonower.com/velours/23378420,23378421,23378422,23378423,23378424,23378433,23378437,23378439,23378441,23378443,23378445,23378447,23378450,23378451,23378452,23378453,23378454,23378455,23378456,23378457?tags=papier,carton,metal,pet_clair,pehd,ela,textiles,verre,organique,dasri,masque,encombrant,autre_non_emballage,environnement,autre,flux_dev,mal_croppe,flou,film_dev_souple,sac_om_plein&datou_id_consolidate=4742&port_consolidate=23377954 Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1361298409, 1361298402, 1361298257] Looping around the photos to save general results len do output : 1 /23377954. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4741', None, None, None, None, None, None, None, '2966278') ('4741', '23377954', '1361298409', None, None, None, None, None, '2966278') ('4741', None, None, None, None, None, None, None, '2966278') ('4741', '23377954', '1361298402', None, None, None, None, None, '2966278') ('4741', None, None, None, None, None, None, None, '2966278') ('4741', '23377954', '1361298257', None, None, None, None, None, '2966278') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 4 time used for this insertion : 0.015012025833129883 save_final save missing photos in datou_result : time spend for datou_step_exec : 4.322456121444702 time spend to save output : 0.015374183654785156 total time spend for step 7 : 4.337830305099487 step8:final Tue May 27 14:27:59 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1361298409: ('0.39634436471898077',), 1361298402: ('0.39634436471898077',), 1361298257: ('0.39634436471898077',)} new output for save of step final : {1361298409: ('0.39634436471898077',), 1361298402: ('0.39634436471898077',), 1361298257: ('0.39634436471898077',)} [1361298409, 1361298402, 1361298257] Looping around the photos to save general results len do output : 3 /1361298409.Didn't retrieve data . /1361298402.Didn't retrieve data . /1361298257.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('4741', None, None, None, None, None, None, None, '2966278') ('4741', '23377954', '1361298409', None, None, None, None, None, '2966278') ('4741', None, None, None, None, None, None, None, '2966278') ('4741', '23377954', '1361298402', None, None, None, None, None, '2966278') ('4741', None, None, None, None, None, None, None, '2966278') ('4741', '23377954', '1361298257', None, None, None, None, None, '2966278') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 9 time used for this insertion : 0.012621164321899414 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.12266325950622559 time spend to save output : 0.012941837310791016 total time spend for step 8 : 0.1356050968170166 step9:velours_tree Tue May 27 14:27:59 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.26981329917907715 time spend to save output : 4.7206878662109375e-05 total time spend for step 9 : 0.26986050605773926 step10:send_mail_cod Tue May 27 14:27:59 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 complete output_args for input 1 complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P23377954_27-05-2025_14_27_59.pdf 23378381 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette233783811748348879 23378382 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette233783821748348881 23378383 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette233783831748348883 23378384 change filename to text .change filename to text .imagette233783841748348884 23378385 imagette233783851748348884 23378386 imagette233783861748348884 23378387 imagette233783871748348884 23378388 imagette233783881748348884 23378389 imagette233783891748348884 23378390 imagette233783901748348884 23378391 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette233783911748348884 23378392 imagette233783921748348884 23378393 imagette233783931748348884 23378394 imagette233783941748348884 23378395 imagette233783951748348884 23378396 imagette233783961748348884 23378397 imagette233783971748348884 23378398 change filename to text .change filename to text .imagette233783981748348885 23378399 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette233783991748348885 23378401 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette233784011748348885 23378402 imagette233784021748348887 23378403 imagette233784031748348887 23378404 imagette233784041748348887 23378405 imagette233784051748348887 23378406 imagette233784061748348887 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=23377954 and hashtag_type = 4855 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/23378420,23378421,23378422,23378423,23378424,23378433,23378437,23378439,23378441,23378443,23378445,23378447,23378450,23378451,23378452,23378453,23378454,23378455,23378456,23378457?tags=papier,carton,metal,pet_clair,pehd,ela,textiles,verre,organique,dasri,masque,encombrant,autre_non_emballage,environnement,autre,flux_dev,mal_croppe,flou,film_dev_souple,sac_om_plein&datou_id_consolidate=4742&port_consolidate=23377954 your option no_mail is active, we will not send the real mail to your client args[1361298409] : ((1361298409, 3180.1069132243583, 492609224), (1361298409, -0.2741192774010967, 496442774), '0.39634436471898077') We are sending mail with results at report@fotonower.com,cod@fotonower.com args[1361298402] : ((1361298402, 1384.9586911260333, 2107751945), (1361298402, -0.4075321606395107, 496442774), '0.39634436471898077') We are sending mail with results at report@fotonower.com,cod@fotonower.com args[1361298257] : ((1361298257, 121.29620009356601, 492688767), (1361298257, -0.16816375780945403, 496442774), '0.39634436471898077') We are sending mail with results at report@fotonower.com,cod@fotonower.com refus_total : 0.39634436471898077 2022-04-13 10:29:59 0 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P23377954_27-05-2025_14_27_59.pdf results_Auto_P23377954_27-05-2025_14_27_59.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P23377954_27-05-2025_14_27_59.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('4741','23377954','results_Auto_P23377954_27-05-2025_14_27_59.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P23377954_27-05-2025_14_27_59.pdf','pdf','','1.3','0.39634436471898077') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1361298409, 1361298402, 1361298257] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('4741', None, None, None, None, None, None, None, '2966278') ('4741', '23377954', '1361298409', None, None, None, None, None, '2966278') ('4741', None, None, None, None, None, None, None, '2966278') ('4741', '23377954', '1361298402', None, None, None, None, None, '2966278') ('4741', None, None, None, None, None, None, None, '2966278') ('4741', '23377954', '1361298257', None, None, None, None, None, '2966278') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 3 time used for this insertion : 0.01202249526977539 save_final save missing photos in datou_result : time spend for datou_step_exec : 9.688576698303223 time spend to save output : 0.012227296829223633 total time spend for step 10 : 9.700803995132446 step11:split_time_score Tue May 27 14:28:09 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('09', 3),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 27052025 23377954 Nombre de photos uploadées : 3 / 23040 (0%) 27052025 23377954 Nombre de photos taguées (types de déchets): 0 / 3 (0%) 27052025 23377954 Nombre de photos taguées (volume) : 0 / 3 (0%) elapsed_time : load_data_split_time_score 1.9073486328125e-06 elapsed_time : order_list_meta_photo_and_scores 5.7220458984375e-06 ??? elapsed_time : fill_and_build_computed_from_old_data 0.0002491474151611328 elapsed_time : insert_dashboard_record_day_entry 0.024565458297729492 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 23377952 order by id desc limit 1 Qualite : 0.39634436471898077 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P23377954_27-05-2025_14_27_59.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 23377954 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 13156 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 13160 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 13160 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 13157 crop_condition is not consistent : 4 used against 2 in the step definition ! WARNING : number of outputs for step 13157 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 13161 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 13159 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 13159 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 13156 doesn't seem to be define in the database( WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 13161 doesn't seem to be define in the database( WARNING : type of input 3 of step 13159 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13163 have datatype=6 whereas input 2 of step 13157 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13162 have datatype=6 whereas input 2 of step 13157 have datatype=None We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 13161 have datatype=10 whereas input 3 of step 13164 have datatype=6 WARNING : output 0 of step 13161 have datatype=10 whereas input 0 of step 13166 have datatype=18 WARNING : type of input 5 of step 13164 doesn't seem to be define in the database( WARNING : output 0 of step 13166 have datatype=11 whereas input 5 of step 13164 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 1 of step 13158 have datatype=7 whereas input 2 of step 13157 have datatype=None WARNING : type of output 3 of step 13157 doesn't seem to be define in the database( WARNING : type of input 3 of step 13161 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13163 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13162 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=23377954 AND mptpi.`type`=4855 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'27052025': {'nb_upload': 3, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1361298409, 1361298402, 1361298257] Looping around the photos to save general results len do output : 1 /23377954Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4741', None, None, None, None, None, None, None, '2966278') ('4741', '23377954', '1361298409', None, None, None, None, None, '2966278') ('4741', None, None, None, None, None, None, None, '2966278') ('4741', '23377954', '1361298402', None, None, None, None, None, '2966278') ('4741', None, None, None, None, None, None, None, '2966278') ('4741', '23377954', '1361298257', None, None, None, None, None, '2966278') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 4 time used for this insertion : 0.013315677642822266 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.1759755611419678 time spend to save output : 0.013526439666748047 total time spend for step 11 : 1.1895020008087158 caffe_path_current : About to save ! 2 After save, about to update current ! ret : 1 len(input) + len(total_photo_id_missing) : 3 set_done_treatment 244.57user 120.20system 6:45.08elapsed 90%CPU (0avgtext+0avgdata 3928580maxresident)k 1129080inputs+309760outputs (2232major+15889812minor)pagefaults 0swaps