python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 8 -a ' -a 4741 -l 30 ' -s datou_current_4741 -M 0 -S 0 -U 95,95,120 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3530087 load datou : 4741 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 13156 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 13160 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 13160 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 13157 crop_condition is not consistent : 4 used against 2 in the step definition ! WARNING : number of outputs for step 13157 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 13161 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 13159 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 13159 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 13156 doesn't seem to be define in the database( WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 13161 doesn't seem to be define in the database( WARNING : type of input 3 of step 13159 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13163 have datatype=6 whereas input 2 of step 13157 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13162 have datatype=6 whereas input 2 of step 13157 have datatype=None We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 13161 have datatype=10 whereas input 3 of step 13164 have datatype=6 WARNING : output 0 of step 13161 have datatype=10 whereas input 0 of step 13166 have datatype=18 WARNING : type of input 5 of step 13164 doesn't seem to be define in the database( WARNING : output 0 of step 13166 have datatype=11 whereas input 5 of step 13164 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 1 of step 13158 have datatype=7 whereas input 2 of step 13157 have datatype=None WARNING : type of output 3 of step 13157 doesn't seem to be define in the database( WARNING : type of input 3 of step 13161 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13163 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13162 doesn't seem to be define in the database( DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5732, 'mask_refus_amiens_050123', 16384, 25088, 'mask_refus_amiens_050123', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 1, 5, 13, 23, 59), datetime.datetime(2023, 1, 5, 13, 23, 59)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5972, 'learn_entrant_syctomXV_111023', 16384, 25088, 'learn_entrant_syctomXV_111023', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 10, 11, 15, 57, 35), datetime.datetime(2023, 10, 11, 15, 57, 35)) load thcls load THCL from format json or kwargs add thcl : 3453 in CacheModelConfig load THCL from format json or kwargs add thcl : 3783 in CacheModelConfig load pdts add pdt : 5732 in CacheModelConfig add pdt : 5972 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 list_input_json: [] Current got : datou_id : 4741, datou_cur_ids : ['2875888'] with mtr_portfolio_ids : ['23056563'] and first list_photo_ids : [] new path : /proc/3530087/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 13156 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 13160 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 13160 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 13157 crop_condition is not consistent : 4 used against 2 in the step definition ! WARNING : number of outputs for step 13157 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 13161 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 13159 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 13159 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 13156 doesn't seem to be define in the database( WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 13161 doesn't seem to be define in the database( WARNING : type of input 3 of step 13159 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13163 have datatype=6 whereas input 2 of step 13157 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13162 have datatype=6 whereas input 2 of step 13157 have datatype=None We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 13161 have datatype=10 whereas input 3 of step 13164 have datatype=6 WARNING : output 0 of step 13161 have datatype=10 whereas input 0 of step 13166 have datatype=18 WARNING : type of input 5 of step 13164 doesn't seem to be define in the database( WARNING : output 0 of step 13166 have datatype=11 whereas input 5 of step 13164 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 1 of step 13158 have datatype=7 whereas input 2 of step 13157 have datatype=None WARNING : type of output 3 of step 13157 doesn't seem to be define in the database( WARNING : type of input 3 of step 13161 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13163 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13162 doesn't seem to be define in the database( DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, mask_detect, blur_detection, brightness, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , BFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 7 ; length of list_pids : 7 ; length of list_args : 7 time to download the photos : 1.4085767269134521 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Fri May 16 11:01:29 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5304 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-05-16 11:01:32.810171: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-16 11:01:32.835247: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-16 11:01:32.837393: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fcc70000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-16 11:01:32.837439: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-16 11:01:32.843285: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-16 11:01:32.992468: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x257a95f0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-16 11:01:32.992518: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-16 11:01:32.993558: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-16 11:01:32.994305: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-16 11:01:32.998969: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-16 11:01:33.002974: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-16 11:01:33.004620: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-16 11:01:33.009072: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-16 11:01:33.011448: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-16 11:01:33.019818: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-16 11:01:33.021541: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-16 11:01:33.021722: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-16 11:01:33.022695: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-16 11:01:33.022740: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-16 11:01:33.022762: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-16 11:01:33.024675: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4843 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-05-16 11:01:33.386508: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-16 11:01:33.386594: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-16 11:01:33.386614: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-16 11:01:33.386633: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-16 11:01:33.386651: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-16 11:01:33.386669: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-16 11:01:33.386687: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-16 11:01:33.386705: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-16 11:01:33.387918: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-16 11:01:33.389142: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-16 11:01:33.389181: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-16 11:01:33.389199: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-16 11:01:33.389215: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-16 11:01:33.389242: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-16 11:01:33.389258: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-16 11:01:33.389274: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-16 11:01:33.389290: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-16 11:01:33.390242: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-16 11:01:33.390277: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-16 11:01:33.390286: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-16 11:01:33.390293: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-16 11:01:33.391320: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4843 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl3783 thcls : [{'id': 3783, 'mtr_user_id': 31, 'name': 'learn_entrant_syctomXV_111023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'photo_desc_type': 5972, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 3783, 'mtr_user_id': 31, 'name': 'learn_entrant_syctomXV_111023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'photo_desc_type': 5972, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5972 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5972, 'learn_entrant_syctomXV_111023', 16384, 25088, 'learn_entrant_syctomXV_111023', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 10, 11, 15, 57, 35), datetime.datetime(2023, 10, 11, 15, 57, 35)) {'thcl': {'id': 3783, 'mtr_user_id': 31, 'name': 'learn_entrant_syctomXV_111023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'photo_desc_type': 5972, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'sac', 'textiles', 'verre', 'organique', 'dasri', 'masque', 'encombrant', 'autre_emballage', 'autre_non_emballage', 'environnement'], 'list_hashtags_csv': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'svm_hashtag_type_desc': 5972, 'photo_desc_type': 5972, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'sac', 'textiles', 'verre', 'organique', 'dasri', 'masque', 'encombrant', 'autre_emballage', 'autre_non_emballage', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_entrant_syctomXV_111023 NUM_CLASSES 11 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_entrant_syctomXV_111023 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-05-16 11:01:44.327023: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-16 11:01:44.523968: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_entrant_syctomXV_111023 /data/models_weight/learn_entrant_syctomXV_111023/mask_model.h5 size_local : 256052544 size in s3 : 256052544 create time local : 2023-10-26 16:36:43 create time in s3 : 2023-10-11 13:57:17 mask_model.h5 already exist and didn't need to update list_images length : 7 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 2.00000 max: 254.00000 molded_images shape: (1, 640, 640, 3) min: -105.73750 max: 128.39687 image_metas shape: (1, 19) min: 0.00000 max: 3120.00000 nb d'objets trouves : 0 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 3.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -118.39141 max: 134.10000 image_metas shape: (1, 19) min: 0.00000 max: 3120.00000 nb d'objets trouves : 1 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 14.00000 max: 231.00000 molded_images shape: (1, 640, 640, 3) min: -97.04609 max: 116.66250 image_metas shape: (1, 19) min: 0.00000 max: 3120.00000 nb d'objets trouves : 0 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 15.00000 max: 232.00000 molded_images shape: (1, 640, 640, 3) min: -93.12578 max: 114.97500 image_metas shape: (1, 19) min: 0.00000 max: 3120.00000 nb d'objets trouves : 0 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 4.00000 max: 244.00000 molded_images shape: (1, 640, 640, 3) min: -118.46172 max: 135.53750 image_metas shape: (1, 19) min: 0.00000 max: 3120.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 19) min: 0.00000 max: 3120.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -121.63750 max: 139.95547 image_metas shape: (1, 19) min: 0.00000 max: 3120.00000 nb d'objets trouves : 2 Detection mask done ! Trying to reset tf kernel 3530251 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 15 tf kernel not reseted sub process len(results) : 7 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 7 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5304 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl3783 Catched exception ! Connect or reconnect ! thcls : [{'id': 3783, 'mtr_user_id': 31, 'name': 'learn_entrant_syctomXV_111023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'photo_desc_type': 5972, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 3783, 'mtr_user_id': 31, 'name': 'learn_entrant_syctomXV_111023', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,sac,textiles,verre,organique,dasri,masque,encombrant,autre_emballage,autre_non_emballage,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4853, 'photo_desc_type': 5972, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5972 ['background', 'sac', 'textiles', 'verre', 'organique', 'dasri', 'masque', 'encombrant', 'autre_emballage', 'autre_non_emballage', 'environnement'] time for calcul the mask position with numpy : 0.3517451286315918 nb_pixel_total : 5190010 time to create 1 rle with new method : 0.7625703811645508 length of segment : 2521 time for calcul the mask position with numpy : 0.15567350387573242 nb_pixel_total : 2104212 time to create 1 rle with new method : 0.13816094398498535 length of segment : 1208 time for calcul the mask position with numpy : 0.14718317985534668 nb_pixel_total : 1819986 time to create 1 rle with new method : 0.3759458065032959 length of segment : 1406 time for calcul the mask position with numpy : 0.010132789611816406 nb_pixel_total : 290680 time to create 1 rle with new method : 0.029752731323242188 length of segment : 1395 time for calcul the mask position with numpy : 0.012689828872680664 nb_pixel_total : 877311 time to create 1 rle with new method : 0.04653787612915039 length of segment : 1023 time for calcul the mask position with numpy : 0.023982524871826172 nb_pixel_total : 1543296 time to create 1 rle with new method : 0.16024017333984375 length of segment : 1585 time for calcul the mask position with numpy : 0.01815032958984375 nb_pixel_total : 980875 time to create 1 rle with new method : 0.1359560489654541 length of segment : 2134 time for calcul the mask position with numpy : 0.5092248916625977 nb_pixel_total : 5470830 time to create 1 rle with new method : 0.5872440338134766 length of segment : 3947 time for calcul the mask position with numpy : 0.24213719367980957 nb_pixel_total : 4092777 time to create 1 rle with new method : 0.2991597652435303 length of segment : 1736 time for calcul the mask position with numpy : 0.011759519577026367 nb_pixel_total : 755853 time to create 1 rle with new method : 0.11519622802734375 length of segment : 1507 time for calcul the mask position with numpy : 0.00462794303894043 nb_pixel_total : 339769 time to create 1 rle with new method : 0.007254600524902344 length of segment : 943 time for calcul the mask position with numpy : 0.015440940856933594 nb_pixel_total : 1062574 time to create 1 rle with new method : 0.16901493072509766 length of segment : 1431 time spent for convertir_results : 10.989296197891235 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 12 chid ids of type : 4854 Number RLEs to save : 0 save missing photos in datou_result : time spend for datou_step_exec : 47.48090052604675 time spend to save output : 0.09197139739990234 total time spend for step 1 : 47.572871923446655 step2:mask_detect Fri May 16 11:02:17 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5304 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-05-16 11:02:20.813313: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-05-16 11:02:20.839172: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-05-16 11:02:20.841374: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fcc74000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-05-16 11:02:20.841426: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-05-16 11:02:20.846624: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-05-16 11:02:20.985821: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x263ae530 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-05-16 11:02:20.985909: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-05-16 11:02:20.987877: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-16 11:02:21.017241: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-16 11:02:21.021676: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-16 11:02:21.027634: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-16 11:02:21.028363: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-16 11:02:21.033307: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-16 11:02:21.035756: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-16 11:02:21.045588: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-16 11:02:21.050146: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-16 11:02:21.050561: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-16 11:02:21.052073: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-16 11:02:21.052105: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-16 11:02:21.052120: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-16 11:02:21.053901: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4843 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-05-16 11:02:21.259816: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-16 11:02:21.259947: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-16 11:02:21.259967: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-16 11:02:21.259985: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-16 11:02:21.260002: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-16 11:02:21.260019: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-16 11:02:21.260035: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-16 11:02:21.260053: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-16 11:02:21.261065: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-16 11:02:21.262342: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-05-16 11:02:21.262388: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-05-16 11:02:21.262405: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-16 11:02:21.262421: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-05-16 11:02:21.262436: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-05-16 11:02:21.262470: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-05-16 11:02:21.262486: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-05-16 11:02:21.262501: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-05-16 11:02:21.263474: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-05-16 11:02:21.263521: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-05-16 11:02:21.263530: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-05-16 11:02:21.263539: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-05-16 11:02:21.264575: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4843 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl3453 thcls : [{'id': 3453, 'mtr_user_id': 31, 'name': 'mask_refus_amiens_050123', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4485, 'photo_desc_type': 5732, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}] thcl {'id': 3453, 'mtr_user_id': 31, 'name': 'mask_refus_amiens_050123', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4485, 'photo_desc_type': 5732, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5732 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5732, 'mask_refus_amiens_050123', 16384, 25088, 'mask_refus_amiens_050123', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 3, datetime.datetime(2023, 1, 5, 13, 23, 59), datetime.datetime(2023, 1, 5, 13, 23, 59)) {'thcl': {'id': 3453, 'mtr_user_id': 31, 'name': 'mask_refus_amiens_050123', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4485, 'photo_desc_type': 5732, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'], 'list_hashtags_csv': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4485, 'svm_hashtag_type_desc': 5732, 'photo_desc_type': 5732, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME mask_refus_amiens_050123 NUM_CLASSES 9 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : mask_refus_amiens_050123 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-05-16 11:02:33.819185: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-05-16 11:02:34.237454: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/mask_refus_amiens_050123 /data/models_weight/mask_refus_amiens_050123/mask_model.h5 size_local : 256009536 size in s3 : 256009536 create time local : 2023-01-05 13:44:07 create time in s3 : 2023-01-05 12:23:52 mask_model.h5 already exist and didn't need to update list_images length : 7 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 3.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -118.39141 max: 134.10000 image_metas shape: (1, 17) min: 0.00000 max: 3120.00000 nb d'objets trouves : 4 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 4.00000 max: 244.00000 molded_images shape: (1, 640, 640, 3) min: -118.46172 max: 135.53750 image_metas shape: (1, 17) min: 0.00000 max: 3120.00000 nb d'objets trouves : 11 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 17) min: 0.00000 max: 3120.00000 nb d'objets trouves : 10 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -121.63750 max: 139.95547 image_metas shape: (1, 17) min: 0.00000 max: 3120.00000 nb d'objets trouves : 11 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 2.00000 max: 254.00000 molded_images shape: (1, 640, 640, 3) min: -105.73750 max: 128.39687 image_metas shape: (1, 17) min: 0.00000 max: 3120.00000 nb d'objets trouves : 1 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 14.00000 max: 231.00000 molded_images shape: (1, 640, 640, 3) min: -97.04609 max: 116.66250 image_metas shape: (1, 17) min: 0.00000 max: 3120.00000 nb d'objets trouves : 2 NEW PHOTO Processing 1 images image shape: (3120, 3120, 3) min: 15.00000 max: 232.00000 molded_images shape: (1, 640, 640, 3) min: -93.12578 max: 114.97500 image_metas shape: (1, 17) min: 0.00000 max: 3120.00000 nb d'objets trouves : 2 Detection mask done ! Trying to reset tf kernel 3532309 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 15 tf kernel not reseted sub process len(results) : 7 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 7 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5304 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl3453 Catched exception ! Connect or reconnect ! thcls : [{'id': 3453, 'mtr_user_id': 31, 'name': 'mask_refus_amiens_050123', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4485, 'photo_desc_type': 5732, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'}] thcl {'id': 3453, 'mtr_user_id': 31, 'name': 'mask_refus_amiens_050123', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,papier,carton,metal,pet_clair,autre,pehd,pet_fonce,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 4485, 'photo_desc_type': 5732, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5732 ['background', 'papier', 'carton', 'metal', 'pet_clair', 'autre', 'pehd', 'pet_fonce', 'environnement'] time for calcul the mask position with numpy : 1.7208971977233887 nb_pixel_total : 4874764 time to create 1 rle with new method : 0.5189132690429688 length of segment : 4478 time for calcul the mask position with numpy : 0.004129648208618164 nb_pixel_total : 378833 time to create 1 rle with new method : 0.009917259216308594 length of segment : 487 time for calcul the mask position with numpy : 0.018687725067138672 nb_pixel_total : 1056550 time to create 1 rle with new method : 0.08039498329162598 length of segment : 678 time for calcul the mask position with numpy : 0.0012977123260498047 nb_pixel_total : 105591 time to create 1 rle with old method : 0.12022829055786133 length of segment : 265 time for calcul the mask position with numpy : 0.002602100372314453 nb_pixel_total : 178359 time to create 1 rle with new method : 0.007236003875732422 length of segment : 942 time for calcul the mask position with numpy : 0.02975177764892578 nb_pixel_total : 697639 time to create 1 rle with new method : 0.4135746955871582 length of segment : 2569 time for calcul the mask position with numpy : 0.0015726089477539062 nb_pixel_total : 111766 time to create 1 rle with old method : 0.14034700393676758 length of segment : 587 time for calcul the mask position with numpy : 0.009016752243041992 nb_pixel_total : 637047 time to create 1 rle with new method : 0.037509918212890625 length of segment : 686 time for calcul the mask position with numpy : 0.013999700546264648 nb_pixel_total : 1073702 time to create 1 rle with new method : 0.12937355041503906 length of segment : 1166 time for calcul the mask position with numpy : 0.009346961975097656 nb_pixel_total : 592619 time to create 1 rle with new method : 0.2524693012237549 length of segment : 1183 time for calcul the mask position with numpy : 0.0064945220947265625 nb_pixel_total : 487534 time to create 1 rle with new method : 0.013233423233032227 length of segment : 647 time for calcul the mask position with numpy : 0.002982616424560547 nb_pixel_total : 230123 time to create 1 rle with new method : 0.007750034332275391 length of segment : 939 time for calcul the mask position with numpy : 0.015493154525756836 nb_pixel_total : 1094748 time to create 1 rle with new method : 0.09370303153991699 length of segment : 1194 time for calcul the mask position with numpy : 0.18454718589782715 nb_pixel_total : 3815123 time to create 1 rle with new method : 0.2374708652496338 length of segment : 1644 time for calcul the mask position with numpy : 0.0067479610443115234 nb_pixel_total : 465853 time to create 1 rle with new method : 0.012835979461669922 length of segment : 605 time for calcul the mask position with numpy : 0.41442155838012695 nb_pixel_total : 5340153 time to create 1 rle with new method : 0.5270013809204102 length of segment : 5614 time for calcul the mask position with numpy : 0.00036025047302246094 nb_pixel_total : 22627 time to create 1 rle with old method : 0.02580881118774414 length of segment : 235 time for calcul the mask position with numpy : 0.015036344528198242 nb_pixel_total : 1144089 time to create 1 rle with new method : 0.07018685340881348 length of segment : 1567 time for calcul the mask position with numpy : 0.0006544589996337891 nb_pixel_total : 39701 time to create 1 rle with old method : 0.056850433349609375 length of segment : 203 time for calcul the mask position with numpy : 0.042836904525756836 nb_pixel_total : 1621441 time to create 1 rle with new method : 0.11327338218688965 length of segment : 840 time for calcul the mask position with numpy : 0.0005784034729003906 nb_pixel_total : 43171 time to create 1 rle with old method : 0.05308413505554199 length of segment : 167 time for calcul the mask position with numpy : 0.008753299713134766 nb_pixel_total : 663559 time to create 1 rle with new method : 0.04449200630187988 length of segment : 1398 time for calcul the mask position with numpy : 0.0001304149627685547 nb_pixel_total : 6101 time to create 1 rle with old method : 0.007344245910644531 length of segment : 61 time for calcul the mask position with numpy : 0.11065840721130371 nb_pixel_total : 2411263 time to create 1 rle with new method : 0.28783297538757324 length of segment : 1434 time for calcul the mask position with numpy : 0.00015354156494140625 nb_pixel_total : 6085 time to create 1 rle with old method : 0.007201194763183594 length of segment : 63 time for calcul the mask position with numpy : 0.14484786987304688 nb_pixel_total : 2943023 time to create 1 rle with new method : 1.097468614578247 length of segment : 3014 time for calcul the mask position with numpy : 0.004712581634521484 nb_pixel_total : 198501 time to create 1 rle with new method : 0.013581991195678711 length of segment : 1346 time for calcul the mask position with numpy : 0.012582063674926758 nb_pixel_total : 1030759 time to create 1 rle with new method : 0.03509926795959473 length of segment : 791 time for calcul the mask position with numpy : 0.0011851787567138672 nb_pixel_total : 32416 time to create 1 rle with old method : 0.03770780563354492 length of segment : 1025 time for calcul the mask position with numpy : 0.0065839290618896484 nb_pixel_total : 490639 time to create 1 rle with new method : 0.0137176513671875 length of segment : 788 time for calcul the mask position with numpy : 0.00045490264892578125 nb_pixel_total : 27488 time to create 1 rle with old method : 0.03271317481994629 length of segment : 145 time for calcul the mask position with numpy : 0.0010449886322021484 nb_pixel_total : 62409 time to create 1 rle with old method : 0.07180261611938477 length of segment : 516 time for calcul the mask position with numpy : 0.0014204978942871094 nb_pixel_total : 83658 time to create 1 rle with old method : 0.09893059730529785 length of segment : 746 time for calcul the mask position with numpy : 0.003039121627807617 nb_pixel_total : 252825 time to create 1 rle with new method : 0.006158590316772461 length of segment : 748 time for calcul the mask position with numpy : 0.0022881031036376953 nb_pixel_total : 179755 time to create 1 rle with new method : 0.004950761795043945 length of segment : 551 time for calcul the mask position with numpy : 0.0003371238708496094 nb_pixel_total : 20311 time to create 1 rle with old method : 0.024164438247680664 length of segment : 103 time for calcul the mask position with numpy : 1.010335922241211 nb_pixel_total : 8314389 time to create 1 rle with new method : 1.4182424545288086 length of segment : 2796 time for calcul the mask position with numpy : 0.8308248519897461 nb_pixel_total : 8740079 time to create 1 rle with new method : 1.0744645595550537 length of segment : 3384 time for calcul the mask position with numpy : 0.014648675918579102 nb_pixel_total : 1245533 time to create 1 rle with new method : 0.06299924850463867 length of segment : 475 time for calcul the mask position with numpy : 0.444504976272583 nb_pixel_total : 8857617 time to create 1 rle with new method : 1.3182332515716553 length of segment : 3175 time for calcul the mask position with numpy : 0.010712146759033203 nb_pixel_total : 528578 time to create 1 rle with new method : 0.2739245891571045 length of segment : 354 time spent for convertir_results : 31.13280940055847 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 53 chid ids of type : 4854 ++++++++++++++++++++++++++++++++++Number RLEs to save : 0 save missing photos in datou_result : time spend for datou_step_exec : 76.11760330200195 time spend to save output : 1.0239431858062744 total time spend for step 2 : 77.14154648780823 step3:blur_detection Fri May 16 11:03:34 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure inside step blur_detection methode: variance laplacian (3120, 3120) 4010.1768528831412 (3120, 3120) 364.7005279992284 (3120, 3120) 51.50154291190086 (3120, 3120) 27.05891574357776 (3120, 3120) 208.18330608719486 (3120, 3120) 922.107306529837 (3120, 3120) 438.2892814472568 Inside saveOutput : final : False verbose : 0 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 7 time used for this insertion : 0.013670206069946289 begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 7 time used for this insertion : 0.013595342636108398 save missing photos in datou_result : time spend for datou_step_exec : 1.0992977619171143 time spend to save output : 0.03667759895324707 total time spend for step 3 : 1.1359753608703613 step4:brightness Fri May 16 11:03:35 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure inside step calcul brightness treat image : temp/1747386088_3530087_1358654261_ccce6f0c84b7b003488ecb74f9212ea9.jpg treat image : temp/1747386088_3530087_1358654238_719804681af1e456a8d0e86d6720a61d.jpg treat image : temp/1747386088_3530087_1358654235_ed5d011fbc4709c4dd4c6f1f3c3b1534.jpg treat image : temp/1747386088_3530087_1358654232_20357fd4058c199b3260e9579c4f56d0.jpg treat image : temp/1747386088_3530087_1358654227_7f5fb8f00650e22cce432d2acac107d3.jpg treat image : temp/1747386088_3530087_1358654223_40b510488d5569a405d9a19a9036995b.jpg treat image : temp/1747386088_3530087_1358654208_33cd3d9fb9b076bee57dfdc1b9b4e669.jpg Inside saveOutput : final : False verbose : 0 begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 7 time used for this insertion : 0.015489578247070312 begin to insert list_values into photo_hahstag_ids : length of list_valuse in save_photo_hashtag_id_type : 7 time used for this insertion : 0.01608586311340332 save missing photos in datou_result : time spend for datou_step_exec : 8.211630582809448 time spend to save output : 0.05101370811462402 total time spend for step 4 : 8.262644290924072 step5:rle_unique_nms_with_priority Fri May 16 11:03:43 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms batch 1 Loaded 53 chid ids of type : 4854 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++nb_obj : 1 nb_hashtags : 1 time to prepare the origin masks : 2.4518134593963623 time for calcul the mask position with numpy : 0.042067766189575195 nb_pixel_total : 1421751 time to create 1 rle with new method : 0.9898803234100342 time for calcul the mask position with numpy : 0.7633571624755859 nb_pixel_total : 8312649 time to create 1 rle with new method : 0.8274548053741455 create new chi : 2.7157938480377197 time to delete rle : 0.04201054573059082 batch 1 Loaded 2 chid ids of type : 4855 Number RLEs to save : 8690 TO DO : save crop sub photo not yet done ! save time : 1.4252855777740479 nb_obj : 5 nb_hashtags : 4 time to prepare the origin masks : 8.882426500320435 time for calcul the mask position with numpy : 0.04237508773803711 nb_pixel_total : 1783147 time to create 1 rle with new method : 0.6609811782836914 time for calcul the mask position with numpy : 0.0325469970703125 nb_pixel_total : 9049 time to create 1 rle with old method : 0.010751962661743164 time for calcul the mask position with numpy : 0.04587388038635254 nb_pixel_total : 1055139 time to create 1 rle with new method : 0.8976054191589355 time for calcul the mask position with numpy : 0.031725168228149414 nb_pixel_total : 378313 time to create 1 rle with new method : 1.654116153717041 time for calcul the mask position with numpy : 0.07415533065795898 nb_pixel_total : 1645653 time to create 1 rle with new method : 1.2364816665649414 time for calcul the mask position with numpy : 0.5299479961395264 nb_pixel_total : 4863099 time to create 1 rle with new method : 1.8323948383331299 create new chi : 7.247126579284668 time to delete rle : 0.0016851425170898438 batch 1 Loaded 6 chid ids of type : 4855 Number RLEs to save : 15163 TO DO : save crop sub photo not yet done ! save time : 2.394395589828491 nb_obj : 2 nb_hashtags : 1 time to prepare the origin masks : 3.331408739089966 time for calcul the mask position with numpy : 0.06230878829956055 nb_pixel_total : 834471 time to create 1 rle with new method : 1.076085090637207 time for calcul the mask position with numpy : 0.051366567611694336 nb_pixel_total : 162772 time to create 1 rle with new method : 0.9409792423248291 time for calcul the mask position with numpy : 0.6773509979248047 nb_pixel_total : 8737157 time to create 1 rle with new method : 1.0900380611419678 create new chi : 4.030714273452759 time to delete rle : 0.001249551773071289 batch 1 Loaded 3 chid ids of type : 4855 Number RLEs to save : 9994 TO DO : save crop sub photo not yet done ! save time : 2.3009378910064697 nb_obj : 2 nb_hashtags : 1 time to prepare the origin masks : 3.468040704727173 time for calcul the mask position with numpy : 0.03487563133239746 nb_pixel_total : 695007 time to create 1 rle with new method : 1.0281822681427002 time for calcul the mask position with numpy : 0.03641772270202637 nb_pixel_total : 192033 time to create 1 rle with new method : 0.6066701412200928 time for calcul the mask position with numpy : 0.4468834400177002 nb_pixel_total : 8847360 time to create 1 rle with new method : 1.2373144626617432 create new chi : 3.5184080600738525 time to delete rle : 0.0011768341064453125 batch 1 Loaded 3 chid ids of type : 4855 Number RLEs to save : 9855 TO DO : save crop sub photo not yet done ! save time : 6.879976511001587 nb_obj : 15 nb_hashtags : 4 time to prepare the origin masks : 17.96203112602234 time for calcul the mask position with numpy : 0.07094430923461914 nb_pixel_total : 1019478 time to create 1 rle with new method : 0.6252529621124268 time for calcul the mask position with numpy : 0.055014848709106445 nb_pixel_total : 8252 time to create 1 rle with old method : 0.01058340072631836 time for calcul the mask position with numpy : 0.07083511352539062 nb_pixel_total : 23439 time to create 1 rle with old method : 0.027104854583740234 time for calcul the mask position with numpy : 0.06065821647644043 nb_pixel_total : 23 time to create 1 rle with old method : 7.939338684082031e-05 time for calcul the mask position with numpy : 0.09183549880981445 nb_pixel_total : 1543811 time to create 1 rle with new method : 0.9491009712219238 time for calcul the mask position with numpy : 0.06752967834472656 nb_pixel_total : 264989 time to create 1 rle with new method : 0.7761845588684082 time for calcul the mask position with numpy : 0.05974125862121582 nb_pixel_total : 141285 time to create 1 rle with old method : 0.15484213829040527 time for calcul the mask position with numpy : 0.06902837753295898 nb_pixel_total : 460923 time to create 1 rle with new method : 1.8722469806671143 time for calcul the mask position with numpy : 0.07128787040710449 nb_pixel_total : 584254 time to create 1 rle with new method : 1.5711123943328857 time for calcul the mask position with numpy : 0.08085870742797852 nb_pixel_total : 1071555 time to create 1 rle with new method : 1.4820377826690674 time for calcul the mask position with numpy : 0.058820247650146484 nb_pixel_total : 70386 time to create 1 rle with old method : 0.07824015617370605 time for calcul the mask position with numpy : 0.06210136413574219 nb_pixel_total : 111643 time to create 1 rle with old method : 0.13359665870666504 time for calcul the mask position with numpy : 0.0923917293548584 nb_pixel_total : 1674607 time to create 1 rle with new method : 1.5933945178985596 time for calcul the mask position with numpy : 0.10629653930664062 nb_pixel_total : 1886500 time to create 1 rle with new method : 1.4988517761230469 time for calcul the mask position with numpy : 0.05711555480957031 nb_pixel_total : 694894 time to create 1 rle with new method : 0.9653410911560059 time for calcul the mask position with numpy : 0.04876112937927246 nb_pixel_total : 178361 time to create 1 rle with new method : 0.8625428676605225 create new chi : 14.126503467559814 time to delete rle : 0.0038368701934814453 batch 1 Loaded 16 chid ids of type : 4855 Number RLEs to save : 28376 TO DO : save crop sub photo not yet done ! save time : 4.066509246826172 nb_obj : 13 nb_hashtags : 3 time to prepare the origin masks : 22.39020824432373 time for calcul the mask position with numpy : 0.06786537170410156 nb_pixel_total : 501374 time to create 1 rle with new method : 1.0973598957061768 time for calcul the mask position with numpy : 0.07568097114562988 nb_pixel_total : 30775 time to create 1 rle with old method : 0.056436777114868164 time for calcul the mask position with numpy : 0.07813715934753418 nb_pixel_total : 557575 time to create 1 rle with new method : 1.5379023551940918 time for calcul the mask position with numpy : 0.07517623901367188 nb_pixel_total : 394913 time to create 1 rle with new method : 1.3531558513641357 time for calcul the mask position with numpy : 0.1259138584136963 nb_pixel_total : 1420104 time to create 1 rle with new method : 1.069688320159912 time for calcul the mask position with numpy : 0.1030416488647461 nb_pixel_total : 175850 time to create 1 rle with new method : 1.0631520748138428 time for calcul the mask position with numpy : 0.06619811058044434 nb_pixel_total : 2751 time to create 1 rle with old method : 0.003223419189453125 time for calcul the mask position with numpy : 0.06462430953979492 nb_pixel_total : 61690 time to create 1 rle with old method : 0.08623623847961426 time for calcul the mask position with numpy : 0.0674283504486084 nb_pixel_total : 83594 time to create 1 rle with old method : 0.09659910202026367 time for calcul the mask position with numpy : 0.05648064613342285 nb_pixel_total : 18419 time to create 1 rle with old method : 0.020105838775634766 time for calcul the mask position with numpy : 0.0633535385131836 nb_pixel_total : 33921 time to create 1 rle with old method : 0.03889131546020508 time for calcul the mask position with numpy : 0.06528544425964355 nb_pixel_total : 318873 time to create 1 rle with new method : 1.1481051445007324 time for calcul the mask position with numpy : 0.05863547325134277 nb_pixel_total : 22533 time to create 1 rle with old method : 0.0254976749420166 time for calcul the mask position with numpy : 0.4998435974121094 nb_pixel_total : 6112028 time to create 1 rle with new method : 0.9891481399536133 create new chi : 10.59231185913086 time to delete rle : 0.003633737564086914 batch 1 Loaded 14 chid ids of type : 4855 Number RLEs to save : 19153 TO DO : save crop sub photo not yet done ! save time : 2.8241312503814697 nb_obj : 13 nb_hashtags : 4 time to prepare the origin masks : 21.58648157119751 time for calcul the mask position with numpy : 0.28630685806274414 nb_pixel_total : 4649092 time to create 1 rle with new method : 0.8490054607391357 time for calcul the mask position with numpy : 0.03072953224182129 nb_pixel_total : 214 time to create 1 rle with old method : 0.00027680397033691406 time for calcul the mask position with numpy : 0.03366494178771973 nb_pixel_total : 3261 time to create 1 rle with old method : 0.0042400360107421875 time for calcul the mask position with numpy : 0.03530764579772949 nb_pixel_total : 9056 time to create 1 rle with old method : 0.010560989379882812 time for calcul the mask position with numpy : 0.033660888671875 nb_pixel_total : 81140 time to create 1 rle with old method : 0.09876179695129395 time for calcul the mask position with numpy : 0.053977012634277344 nb_pixel_total : 42727 time to create 1 rle with old method : 0.050786495208740234 time for calcul the mask position with numpy : 0.05246591567993164 nb_pixel_total : 5620 time to create 1 rle with old method : 0.006627321243286133 time for calcul the mask position with numpy : 0.05438995361328125 nb_pixel_total : 376596 time to create 1 rle with new method : 0.9815053939819336 time for calcul the mask position with numpy : 0.06555318832397461 nb_pixel_total : 32224 time to create 1 rle with old method : 0.039449214935302734 time for calcul the mask position with numpy : 0.061467885971069336 nb_pixel_total : 142184 time to create 1 rle with old method : 0.16124248504638672 time for calcul the mask position with numpy : 0.06669855117797852 nb_pixel_total : 198379 time to create 1 rle with new method : 1.1652967929840088 time for calcul the mask position with numpy : 0.05694842338562012 nb_pixel_total : 906657 time to create 1 rle with new method : 0.6973426342010498 time for calcul the mask position with numpy : 0.0570530891418457 nb_pixel_total : 346268 time to create 1 rle with new method : 0.6664419174194336 time for calcul the mask position with numpy : 0.09012532234191895 nb_pixel_total : 2940982 time to create 1 rle with new method : 0.633692741394043 create new chi : 6.606595039367676 time to delete rle : 0.0032737255096435547 batch 1 Loaded 14 chid ids of type : 4855 Number RLEs to save : 21725 TO DO : save crop sub photo not yet done ! save time : 3.6108689308166504 map_output_result : {1358654261: (0.40843746077883686, 'Should be the crop_list due to order', 0), 1358654238: (0.40843746077883686, 'Should be the crop_list due to order', 0.9955489752903938), 1358654235: (0.40843746077883686, 'Should be the crop_list due to order', 0), 1358654232: (0.40843746077883686, 'Should be the crop_list due to order', 0), 1358654227: (0.40843746077883686, 'Should be the crop_list due to order', 0.6921381585015514), 1358654223: (0.40843746077883686, 'Should be the crop_list due to order', 1.0), 1358654208: (0.40843746077883686, 'Should be the crop_list due to order', 0.17137509165991277)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1358654261, 1358654238, 1358654235, 1358654232, 1358654227, 1358654223, 1358654208] Looping around the photos to save general results len do output : 7 /1358654261.Didn't retrieve data . /1358654238.Didn't retrieve data . /1358654235.Didn't retrieve data . /1358654232.Didn't retrieve data . /1358654227.Didn't retrieve data . /1358654223.Didn't retrieve data . /1358654208.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654261', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654238', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654235', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654232', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654227', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654223', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654208', None, None, None, None, None, '2875888') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 21 time used for this insertion : 0.04104423522949219 save_final save missing photos in datou_result : time spend for datou_step_exec : 153.91977071762085 time spend to save output : 0.06004834175109863 total time spend for step 5 : 153.97981905937195 step6:crop_condition Fri May 16 11:06:17 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 4855 Loading chi in step crop for list_pids : 7 ! batch 1 Loaded 58 chid ids of type : 4855 begin to crop the class : papier param for this class : {'min_score': 0.1} filtre for class : papier hashtag_id of this class : 492668766 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 3015255 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1747386388_3530087 we have uploaded 2 photos in the portfolio 3015255 time of upload the photos Elapsed time : 1.2939445972442627 we have finished the crop for the class : papier begin to crop the class : carton param for this class : {'min_score': 0.1} filtre for class : carton hashtag_id of this class : 492774966 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 3015255 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1747386390_3530087 we have uploaded 1 photos in the portfolio 3015255 time of upload the photos Elapsed time : 0.5549540519714355 we have finished the crop for the class : carton begin to crop the class : metal param for this class : {'min_score': 0.1} filtre for class : metal hashtag_id of this class : 492628673 begin to crop the class : pet_clair param for this class : {'min_score': 0.1} filtre for class : pet_clair hashtag_id of this class : 2107755846 begin to crop the class : pehd param for this class : {'min_score': 0.1} filtre for class : pehd hashtag_id of this class : 628944319 begin to crop the class : pet_fonce param for this class : {'min_score': 0.1} filtre for class : pet_fonce hashtag_id of this class : 2107755900 begin to crop the class : sac param for this class : {'min_score': 0.1} filtre for class : sac hashtag_id of this class : 492618350 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 3015255 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1747386393_3530087 we have uploaded 1 photos in the portfolio 3015255 time of upload the photos Elapsed time : 0.528411865234375 we have finished the crop for the class : sac begin to crop the class : textiles param for this class : {'min_score': 0.1} filtre for class : textiles hashtag_id of this class : 494789423 begin to crop the class : verre param for this class : {'min_score': 0.1} filtre for class : verre hashtag_id of this class : 492631919 begin to crop the class : organique param for this class : {'min_score': 0.1} filtre for class : organique hashtag_id of this class : 530726390 begin to crop the class : dasri param for this class : {'min_score': 0.1} filtre for class : dasri hashtag_id of this class : 1608424396 begin to crop the class : masque param for this class : {'min_score': 0.1} filtre for class : masque hashtag_id of this class : 492626781 begin to crop the class : encombrant param for this class : {'min_score': 0.1} filtre for class : encombrant hashtag_id of this class : 605975549 begin to crop the class : autre param for this class : {'min_score': 0.1} filtre for class : autre hashtag_id of this class : 494826614 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 3015255 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1747386403_3530087 we have uploaded 3 photos in the portfolio 3015255 time of upload the photos Elapsed time : 1.4089133739471436 we have finished the crop for the class : autre begin to crop the class : autre_emballage param for this class : {'min_score': 0.1} filtre for class : autre_emballage hashtag_id of this class : 2107760328 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 3015255 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1747386413_3530087 we have uploaded 1 photos in the portfolio 3015255 time of upload the photos Elapsed time : 0.6828575134277344 we have finished the crop for the class : autre_emballage begin to crop the class : autre_non_emballage param for this class : {'min_score': 0.1} filtre for class : autre_non_emballage hashtag_id of this class : 2107760329 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 3015255 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1747386423_3530087 we have uploaded 1 photos in the portfolio 3015255 time of upload the photos Elapsed time : 0.8585803508758545 we have finished the crop for the class : autre_non_emballage begin to crop the class : environnement param for this class : {'min_score': 0.1} filtre for class : environnement hashtag_id of this class : 493012381 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 22 About to insert : list_path_to_insert length 22 new photo from crops ! About to upload 22 photos upload in portfolio : 3015255 init cache_photo without model_param we have 22 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1747386649_3530087 we have uploaded 22 photos in the portfolio 3015255 time of upload the photos Elapsed time : 8.089743375778198 we have finished the crop for the class : environnement delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1358654261, 1358654238, 1358654235, 1358654232, 1358654227, 1358654223, 1358654208] Looping around the photos to save general results len do output : 31 /1358899944Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358899946Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358899959Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358899981Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358900051Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358900052Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358900053Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358900084Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358900096Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358900268Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358900269Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358900270Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358900271Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358900272Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358900273Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358900274Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358900275Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358900276Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358900277Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358900278Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358900279Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358900280Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358900282Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358900283Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358900285Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358900286Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358900287Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358900288Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358900289Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358900290Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1358900291Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654261', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654238', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654235', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654232', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654227', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654223', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654208', None, None, None, None, None, '2875888') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 100 time used for this insertion : 0.321850061416626 save_final save missing photos in datou_result : time spend for datou_step_exec : 284.717661857605 time spend to save output : 0.8770241737365723 total time spend for step 6 : 285.59468603134155 step7:ventilate_hashtags_in_portfolio Fri May 16 11:11:03 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! Iterating over portfolio : 23056563 get user id for portfolio 23056563 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=23056563 AND mptpi.`type`=4855 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','carton','metal','pet_clair','pehd','pet_fonce','pet_opaque','barquette_opaque','film_plastique','ela','sac','textiles','verre','organique','dasri','masque','encombrant','autre_emballage','autre_non_emballage','environnement','autre','flux_dev','mal_croppe','flou','film_dev_souple','sac_om_plein')) AND mptpi.`min_score`=0.1 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=23056563 AND mptpi.`type`=4855 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','carton','metal','pet_clair','pehd','pet_fonce','pet_opaque','barquette_opaque','film_plastique','ela','sac','textiles','verre','organique','dasri','masque','encombrant','autre_emballage','autre_non_emballage','environnement','autre','flux_dev','mal_croppe','flou','film_dev_souple','sac_om_plein')) AND mptpi.`min_score`=0.1 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=23056563 AND mptpi.`type`=4856 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','carton','metal','pet_clair','pehd','pet_fonce','pet_opaque','barquette_opaque','film_plastique','ela','sac','textiles','verre','organique','dasri','masque','encombrant','autre_emballage','autre_non_emballage','environnement','autre','flux_dev','mal_croppe','flou','film_dev_souple','sac_om_plein')) AND mptpi.`min_score`=0.1 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=23056563 AND mptpi.`type`=4856 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('papier','carton','metal','pet_clair','pehd','pet_fonce','pet_opaque','barquette_opaque','film_plastique','ela','sac','textiles','verre','organique','dasri','masque','encombrant','autre_emballage','autre_non_emballage','environnement','autre','flux_dev','mal_croppe','flou','film_dev_souple','sac_om_plein')) AND mptpi.`min_score`=0.1 To do lien utilise dans velours : https://www.fotonower.com/velours/23058176,23058177,23058178,23058179,23058180,23058185,23058187,23058188,23058189,23058190,23058191,23058192,23058194,23058195,23058196,23058197,23058198,23058199,23058200,23058201?tags=papier,carton,metal,pet_clair,pehd,ela,textiles,verre,organique,dasri,masque,encombrant,autre_non_emballage,environnement,autre,flux_dev,mal_croppe,flou,film_dev_souple,sac_om_plein&datou_id_consolidate=4742&port_consolidate=23056563 Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1358654261, 1358654238, 1358654235, 1358654232, 1358654227, 1358654223, 1358654208] Looping around the photos to save general results len do output : 1 /23056563. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654261', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654238', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654235', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654232', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654227', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654223', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654208', None, None, None, None, None, '2875888') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 8 time used for this insertion : 0.022469282150268555 save_final save missing photos in datou_result : time spend for datou_step_exec : 4.76340389251709 time spend to save output : 0.02337050437927246 total time spend for step 7 : 4.786774396896362 step8:final Fri May 16 11:11:08 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1358654261: ('0.7890728070575697',), 1358654238: ('0.7890728070575697',), 1358654235: ('0.7890728070575697',), 1358654232: ('0.7890728070575697',), 1358654227: ('0.7890728070575697',), 1358654223: ('0.7890728070575697',), 1358654208: ('0.7890728070575697',)} new output for save of step final : {1358654261: ('0.7890728070575697',), 1358654238: ('0.7890728070575697',), 1358654235: ('0.7890728070575697',), 1358654232: ('0.7890728070575697',), 1358654227: ('0.7890728070575697',), 1358654223: ('0.7890728070575697',), 1358654208: ('0.7890728070575697',)} [1358654261, 1358654238, 1358654235, 1358654232, 1358654227, 1358654223, 1358654208] Looping around the photos to save general results len do output : 7 /1358654261.Didn't retrieve data . /1358654238.Didn't retrieve data . /1358654235.Didn't retrieve data . /1358654232.Didn't retrieve data . /1358654227.Didn't retrieve data . /1358654223.Didn't retrieve data . /1358654208.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654261', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654238', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654235', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654232', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654227', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654223', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654208', None, None, None, None, None, '2875888') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 21 time used for this insertion : 0.06414008140563965 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.2397305965423584 time spend to save output : 0.06479263305664062 total time spend for step 8 : 0.304523229598999 step9:velours_tree Fri May 16 11:11:08 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 2.013566493988037 time spend to save output : 5.078315734863281e-05 total time spend for step 9 : 2.0136172771453857 step10:send_mail_cod Fri May 16 11:11:10 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 complete output_args for input 1 complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P23056563_16-05-2025_11_11_10.pdf 23058149 change filename to text .change filename to text .imagette230581491747386670 23058150 change filename to text .imagette230581501747386670 23058151 imagette230581511747386670 23058152 imagette230581521747386670 23058153 imagette230581531747386670 23058154 imagette230581541747386670 23058155 imagette230581551747386670 23058156 imagette230581561747386670 23058157 imagette230581571747386670 23058158 imagette230581581747386670 23058159 change filename to text .imagette230581591747386670 23058160 imagette230581601747386670 23058161 imagette230581611747386671 23058162 imagette230581621747386671 23058163 imagette230581631747386671 23058164 imagette230581641747386671 23058165 imagette230581651747386671 23058166 change filename to text .imagette230581661747386671 23058167 change filename to text .imagette230581671747386671 23058169 change filename to text .change filename to text .change filename to text .imagette230581691747386671 23058170 imagette230581701747386671 23058171 imagette230581711747386671 23058172 imagette230581721747386671 23058173 imagette230581731747386671 23058174 imagette230581741747386671 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=23056563 and hashtag_type = 4855 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/23058176,23058177,23058178,23058179,23058180,23058185,23058187,23058188,23058189,23058190,23058191,23058192,23058194,23058195,23058196,23058197,23058198,23058199,23058200,23058201?tags=papier,carton,metal,pet_clair,pehd,ela,textiles,verre,organique,dasri,masque,encombrant,autre_non_emballage,environnement,autre,flux_dev,mal_croppe,flou,film_dev_souple,sac_om_plein&datou_id_consolidate=4742&port_consolidate=23056563 your option no_mail is active, we will not send the real mail to your client args[1358654261] : ((1358654261, 4010.1768528831412, 492609224), (1358654261, 1.275503686441946, 2107752395), '0.7890728070575697') We are sending mail with results at report@fotonower.com,cod@fotonower.com args[1358654238] : ((1358654238, 364.7005279992284, 492688767), (1358654238, -0.008930115258630307, 2107752395), '0.7890728070575697') We are sending mail with results at report@fotonower.com,cod@fotonower.com args[1358654235] : ((1358654235, 51.50154291190086, 492688767), (1358654235, 1.470802148906138, 2107752395), '0.7890728070575697') We are sending mail with results at report@fotonower.com,cod@fotonower.com args[1358654232] : ((1358654232, 27.05891574357776, 492688767), (1358654232, 1.388348134372454, 2107752395), '0.7890728070575697') We are sending mail with results at report@fotonower.com,cod@fotonower.com args[1358654227] : ((1358654227, 208.18330608719486, 492688767), (1358654227, 0.07828510202416934, 2107752395), '0.7890728070575697') We are sending mail with results at report@fotonower.com,cod@fotonower.com args[1358654223] : ((1358654223, 922.107306529837, 2107751945), (1358654223, -0.4570736032478558, 496442774), '0.7890728070575697') We are sending mail with results at report@fotonower.com,cod@fotonower.com args[1358654208] : ((1358654208, 438.2892814472568, 492688767), (1358654208, -0.2709348472032176, 496442774), '0.7890728070575697') We are sending mail with results at report@fotonower.com,cod@fotonower.com refus_total : 0.7890728070575697 2022-04-13 10:29:59 0 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P23056563_16-05-2025_11_11_10.pdf results_Auto_P23056563_16-05-2025_11_11_10.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P23056563_16-05-2025_11_11_10.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('4741','23056563','results_Auto_P23056563_16-05-2025_11_11_10.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P23056563_16-05-2025_11_11_10.pdf','pdf','','0.78','0.7890728070575697') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1358654261, 1358654238, 1358654235, 1358654232, 1358654227, 1358654223, 1358654208] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654261', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654238', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654235', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654232', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654227', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654223', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654208', None, None, None, None, None, '2875888') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 7 time used for this insertion : 0.030109882354736328 save_final save missing photos in datou_result : time spend for datou_step_exec : 3.912088394165039 time spend to save output : 0.030339479446411133 total time spend for step 10 : 3.94242787361145 step11:split_time_score Fri May 16 11:11:14 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('24', 1), ('27', 1), ('28', 1), ('34', 4)) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 15052025 23056563 Nombre de photos uploadées : 7 / 23040 (0%) 15052025 23056563 Nombre de photos taguées (types de déchets): 0 / 7 (0%) 15052025 23056563 Nombre de photos taguées (volume) : 0 / 7 (0%) elapsed_time : load_data_split_time_score 2.6226043701171875e-06 elapsed_time : order_list_meta_photo_and_scores 6.198883056640625e-06 ??????? elapsed_time : fill_and_build_computed_from_old_data 0.0003631114959716797 elapsed_time : insert_dashboard_record_day_entry 0.028825759887695312 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.7890728070575697 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P23056563_16-05-2025_11_11_10.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 23056563 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 13156 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 13160 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 13160 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 13157 crop_condition is not consistent : 4 used against 2 in the step definition ! WARNING : number of outputs for step 13157 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 13161 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 13159 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 13159 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 13156 doesn't seem to be define in the database( WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 13161 doesn't seem to be define in the database( WARNING : type of input 3 of step 13159 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13163 have datatype=6 whereas input 2 of step 13157 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13162 have datatype=6 whereas input 2 of step 13157 have datatype=None We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 13161 have datatype=10 whereas input 3 of step 13164 have datatype=6 WARNING : output 0 of step 13161 have datatype=10 whereas input 0 of step 13166 have datatype=18 WARNING : type of input 5 of step 13164 doesn't seem to be define in the database( WARNING : output 0 of step 13166 have datatype=11 whereas input 5 of step 13164 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 1 of step 13158 have datatype=7 whereas input 2 of step 13157 have datatype=None WARNING : type of output 3 of step 13157 doesn't seem to be define in the database( WARNING : type of input 3 of step 13161 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13163 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13162 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=23056563 AND mptpi.`type`=4855 To do Qualite : 1.0 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P23056565_16-05-2025_10_30_10.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 23056565 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 13156 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 13160 mask_detect have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 13160 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 13158 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 13157 crop_condition is not consistent : 4 used against 2 in the step definition ! WARNING : number of outputs for step 13157 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 13161 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 13159 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 13159 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 13156 doesn't seem to be define in the database( WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 13161 doesn't seem to be define in the database( WARNING : type of input 3 of step 13159 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13163 have datatype=6 whereas input 2 of step 13157 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 0 of step 13162 have datatype=6 whereas input 2 of step 13157 have datatype=None We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 13161 have datatype=10 whereas input 3 of step 13164 have datatype=6 WARNING : output 0 of step 13161 have datatype=10 whereas input 0 of step 13166 have datatype=18 WARNING : type of input 5 of step 13164 doesn't seem to be define in the database( WARNING : output 0 of step 13166 have datatype=11 whereas input 5 of step 13164 have datatype=None WARNING : type of input 2 of step 13157 doesn't seem to be define in the database( WARNING : output 1 of step 13158 have datatype=7 whereas input 2 of step 13157 have datatype=None WARNING : type of output 3 of step 13157 doesn't seem to be define in the database( WARNING : type of input 3 of step 13161 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13163 doesn't seem to be define in the database( WARNING : type of output 2 of step 13160 doesn't seem to be define in the database( WARNING : type of input 1 of step 13162 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=23056565 AND mptpi.`type`=4855 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'15052025': {'nb_upload': 7, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1358654261, 1358654238, 1358654235, 1358654232, 1358654227, 1358654223, 1358654208] Looping around the photos to save general results len do output : 1 /23056563Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654261', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654238', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654235', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654232', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654227', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654223', None, None, None, None, None, '2875888') ('4741', None, None, None, None, None, None, None, '2875888') ('4741', '23056563', '1358654208', None, None, None, None, None, '2875888') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 8 time used for this insertion : 0.04069161415100098 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.3229851722717285 time spend to save output : 0.040914058685302734 total time spend for step 11 : 0.36389923095703125 caffe_path_current : About to save ! 2 After save, about to update current ! ret : 1 len(input) + len(total_photo_id_missing) : 7 set_done_treatment 354.56user 184.08system 9:48.44elapsed 91%CPU (0avgtext+0avgdata 3347424maxresident)k 2061768inputs+413624outputs (24624major+15579721minor)pagefaults 0swaps