python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 3348977 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 14>10 please execute split_portfolio.py -i 20476171 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2579320'] with mtr_portfolio_ids : ['20476171'] and first list_photo_ids : [] new path : /proc/3348977/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 2.4417531490325928 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Wed Feb 12 12:24:32 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 2900 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-12 12:24:35.020830: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-12 12:24:35.032316: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-12 12:24:35.033787: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f870c000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-12 12:24:35.033825: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-12 12:24:35.036052: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-12 12:24:35.267595: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x43648bd0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-12 12:24:35.267641: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-12 12:24:35.268683: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-12 12:24:35.269035: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-12 12:24:35.271966: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-12 12:24:35.274765: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-12 12:24:35.275263: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-12 12:24:35.277976: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-12 12:24:35.279213: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-12 12:24:35.284131: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-12 12:24:35.285291: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-12 12:24:35.285358: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-12 12:24:35.285995: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-12 12:24:35.286014: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-12 12:24:35.286025: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-12 12:24:35.287181: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2448 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-12 12:24:35.558323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-12 12:24:35.558517: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-12 12:24:35.558559: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-12 12:24:35.558587: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-12 12:24:35.558626: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-12 12:24:35.558655: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-12 12:24:35.558685: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-12 12:24:35.558714: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-12 12:24:35.560030: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-12 12:24:35.561024: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-12 12:24:35.561063: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-12 12:24:35.561081: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-12 12:24:35.561098: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-12 12:24:35.561114: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-12 12:24:35.561131: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-12 12:24:35.561149: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-12 12:24:35.561166: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-12 12:24:35.561954: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-12 12:24:35.561995: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-12 12:24:35.562003: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-12 12:24:35.562012: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-12 12:24:35.562842: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2448 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-12 12:24:47.574838: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-12 12:24:47.751822: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-12 12:24:49.226845: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-12 12:24:49.226922: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-12 12:24:49.233525: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-12 12:24:49.233554: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-12 12:24:49.288402: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-12 12:24:49.288469: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.06GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-12 12:24:49.329718: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.09GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-12 12:24:49.329792: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.09GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-12 12:24:49.377755: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.15GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-12 12:24:49.377811: W tensorflow/core/common_runtime/bfc_allocator.cc:245] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.15GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2025-02-12 12:24:49.379983: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.39G (1495138304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.380518: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.25G (1345624576 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.702085: W tensorflow/core/common_runtime/bfc_allocator.cc:311] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature. 2025-02-12 12:24:49.744299: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.744969: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.745872: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.746392: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.757279: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.757801: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.758338: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.758855: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.759408: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.759967: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.791144: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.791744: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.816549: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.816611: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2025-02-12 12:24:49.817269: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.817985: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.824938: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.825481: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.833331: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.833854: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.852048: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.852655: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.853229: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.853788: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.857836: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.858385: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.859100: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.859667: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.860667: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.871359: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.872289: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.881426: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.882013: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.882584: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.883170: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.883744: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory 2025-02-12 12:24:49.884302: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 1.89G (2032009216 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 18 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 9 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 19 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 22 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 20 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 16 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 5 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 16 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 18 NEW PHOTO Processing 1 images image shape: (2160, 3840, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3840.00000 nb d'objets trouves : 19 Detection mask done ! Trying to reset tf kernel 3349567 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 2149 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 3342 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.0053272247314453125 nb_pixel_total : 178874 time to create 1 rle with new method : 0.011594295501708984 length of segment : 598 time for calcul the mask position with numpy : 0.0019402503967285156 nb_pixel_total : 118945 time to create 1 rle with old method : 0.13269424438476562 length of segment : 430 time for calcul the mask position with numpy : 0.0018298625946044922 nb_pixel_total : 128343 time to create 1 rle with old method : 0.14279389381408691 length of segment : 395 time for calcul the mask position with numpy : 0.003286123275756836 nb_pixel_total : 152536 time to create 1 rle with new method : 0.01058340072631836 length of segment : 452 time for calcul the mask position with numpy : 0.001922607421875 nb_pixel_total : 126535 time to create 1 rle with old method : 0.14797282218933105 length of segment : 378 time for calcul the mask position with numpy : 0.0030944347381591797 nb_pixel_total : 180025 time to create 1 rle with new method : 0.009668111801147461 length of segment : 1566 time for calcul the mask position with numpy : 0.029952049255371094 nb_pixel_total : 1570534 time to create 1 rle with new method : 0.08863377571105957 length of segment : 2430 time for calcul the mask position with numpy : 0.0005333423614501953 nb_pixel_total : 24359 time to create 1 rle with old method : 0.03003835678100586 length of segment : 131 time for calcul the mask position with numpy : 0.0020406246185302734 nb_pixel_total : 118005 time to create 1 rle with old method : 0.14026474952697754 length of segment : 396 time for calcul the mask position with numpy : 0.007833480834960938 nb_pixel_total : 372049 time to create 1 rle with new method : 0.021658897399902344 length of segment : 1487 time for calcul the mask position with numpy : 0.0012574195861816406 nb_pixel_total : 40369 time to create 1 rle with old method : 0.06894707679748535 length of segment : 173 time for calcul the mask position with numpy : 0.0012166500091552734 nb_pixel_total : 41469 time to create 1 rle with old method : 0.04646611213684082 length of segment : 236 time for calcul the mask position with numpy : 0.0029540061950683594 nb_pixel_total : 94030 time to create 1 rle with old method : 0.1132056713104248 length of segment : 409 time for calcul the mask position with numpy : 0.0007226467132568359 nb_pixel_total : 41443 time to create 1 rle with old method : 0.04788637161254883 length of segment : 161 time for calcul the mask position with numpy : 0.0011110305786132812 nb_pixel_total : 33315 time to create 1 rle with old method : 0.03913068771362305 length of segment : 189 time for calcul the mask position with numpy : 0.001016855239868164 nb_pixel_total : 22000 time to create 1 rle with old method : 0.025783061981201172 length of segment : 302 time for calcul the mask position with numpy : 0.003668069839477539 nb_pixel_total : 146111 time to create 1 rle with old method : 0.16274285316467285 length of segment : 477 time for calcul the mask position with numpy : 0.0032525062561035156 nb_pixel_total : 177432 time to create 1 rle with new method : 0.00454401969909668 length of segment : 597 time for calcul the mask position with numpy : 0.5578200817108154 nb_pixel_total : 3528113 time to create 1 rle with new method : 0.580512523651123 length of segment : 2584 time for calcul the mask position with numpy : 0.0006730556488037109 nb_pixel_total : 45555 time to create 1 rle with old method : 0.051244497299194336 length of segment : 187 time for calcul the mask position with numpy : 0.0021839141845703125 nb_pixel_total : 130043 time to create 1 rle with old method : 0.16603493690490723 length of segment : 1257 time for calcul the mask position with numpy : 0.018473148345947266 nb_pixel_total : 1329441 time to create 1 rle with new method : 0.04686450958251953 length of segment : 2647 time for calcul the mask position with numpy : 0.0015027523040771484 nb_pixel_total : 55173 time to create 1 rle with old method : 0.06148028373718262 length of segment : 253 time for calcul the mask position with numpy : 0.02444911003112793 nb_pixel_total : 987893 time to create 1 rle with new method : 0.052275657653808594 length of segment : 1733 time for calcul the mask position with numpy : 0.0178830623626709 nb_pixel_total : 757490 time to create 1 rle with new method : 0.029870033264160156 length of segment : 1432 time for calcul the mask position with numpy : 0.0005660057067871094 nb_pixel_total : 11781 time to create 1 rle with old method : 0.01319742202758789 length of segment : 141 time for calcul the mask position with numpy : 0.004445314407348633 nb_pixel_total : 150115 time to create 1 rle with new method : 0.006425380706787109 length of segment : 577 time for calcul the mask position with numpy : 0.002032041549682617 nb_pixel_total : 68381 time to create 1 rle with old method : 0.07972884178161621 length of segment : 423 time for calcul the mask position with numpy : 0.003114461898803711 nb_pixel_total : 95078 time to create 1 rle with old method : 0.10474705696105957 length of segment : 369 time for calcul the mask position with numpy : 0.000164031982421875 nb_pixel_total : 7294 time to create 1 rle with old method : 0.00789022445678711 length of segment : 91 time for calcul the mask position with numpy : 0.015513896942138672 nb_pixel_total : 670163 time to create 1 rle with new method : 0.028661251068115234 length of segment : 1524 time for calcul the mask position with numpy : 0.0005419254302978516 nb_pixel_total : 15481 time to create 1 rle with old method : 0.017728805541992188 length of segment : 119 time for calcul the mask position with numpy : 0.0023713111877441406 nb_pixel_total : 74215 time to create 1 rle with old method : 0.08253240585327148 length of segment : 302 time for calcul the mask position with numpy : 0.0016903877258300781 nb_pixel_total : 36839 time to create 1 rle with old method : 0.04027390480041504 length of segment : 519 time for calcul the mask position with numpy : 0.0024530887603759766 nb_pixel_total : 171119 time to create 1 rle with new method : 0.006621599197387695 length of segment : 577 time for calcul the mask position with numpy : 0.0003070831298828125 nb_pixel_total : 14519 time to create 1 rle with old method : 0.016785144805908203 length of segment : 110 time for calcul the mask position with numpy : 0.0005786418914794922 nb_pixel_total : 18901 time to create 1 rle with old method : 0.021335601806640625 length of segment : 151 time for calcul the mask position with numpy : 0.0024480819702148438 nb_pixel_total : 181690 time to create 1 rle with new method : 0.006260871887207031 length of segment : 1080 time for calcul the mask position with numpy : 0.01609349250793457 nb_pixel_total : 705927 time to create 1 rle with new method : 0.04036998748779297 length of segment : 1746 time for calcul the mask position with numpy : 0.0027680397033691406 nb_pixel_total : 56901 time to create 1 rle with old method : 0.06238150596618652 length of segment : 625 time for calcul the mask position with numpy : 0.0006906986236572266 nb_pixel_total : 18029 time to create 1 rle with old method : 0.02364826202392578 length of segment : 159 time for calcul the mask position with numpy : 0.004652500152587891 nb_pixel_total : 173067 time to create 1 rle with new method : 0.005969047546386719 length of segment : 625 time for calcul the mask position with numpy : 0.0004467964172363281 nb_pixel_total : 9804 time to create 1 rle with old method : 0.011191606521606445 length of segment : 96 time for calcul the mask position with numpy : 0.0009953975677490234 nb_pixel_total : 22695 time to create 1 rle with old method : 0.028775691986083984 length of segment : 201 time for calcul the mask position with numpy : 0.00016355514526367188 nb_pixel_total : 2153 time to create 1 rle with old method : 0.0026671886444091797 length of segment : 47 time for calcul the mask position with numpy : 0.002951383590698242 nb_pixel_total : 62491 time to create 1 rle with old method : 0.06663322448730469 length of segment : 423 time for calcul the mask position with numpy : 0.004517316818237305 nb_pixel_total : 163688 time to create 1 rle with new method : 0.006361722946166992 length of segment : 452 time for calcul the mask position with numpy : 0.0006232261657714844 nb_pixel_total : 23566 time to create 1 rle with old method : 0.02677464485168457 length of segment : 235 time for calcul the mask position with numpy : 0.0056378841400146484 nb_pixel_total : 197607 time to create 1 rle with new method : 0.007546186447143555 length of segment : 747 time for calcul the mask position with numpy : 0.00017952919006347656 nb_pixel_total : 1799 time to create 1 rle with old method : 0.0022916793823242188 length of segment : 61 time for calcul the mask position with numpy : 0.000881195068359375 nb_pixel_total : 21853 time to create 1 rle with old method : 0.02495265007019043 length of segment : 137 time for calcul the mask position with numpy : 0.0013723373413085938 nb_pixel_total : 48077 time to create 1 rle with old method : 0.05321693420410156 length of segment : 341 time for calcul the mask position with numpy : 0.00017762184143066406 nb_pixel_total : 2284 time to create 1 rle with old method : 0.002629518508911133 length of segment : 78 time for calcul the mask position with numpy : 0.017195463180541992 nb_pixel_total : 654135 time to create 1 rle with new method : 0.028992414474487305 length of segment : 1371 time for calcul the mask position with numpy : 0.0006155967712402344 nb_pixel_total : 13175 time to create 1 rle with old method : 0.014369964599609375 length of segment : 169 time for calcul the mask position with numpy : 0.00017404556274414062 nb_pixel_total : 2074 time to create 1 rle with old method : 0.0024318695068359375 length of segment : 68 time for calcul the mask position with numpy : 0.002424001693725586 nb_pixel_total : 78213 time to create 1 rle with old method : 0.08421587944030762 length of segment : 365 time for calcul the mask position with numpy : 0.000732421875 nb_pixel_total : 19134 time to create 1 rle with old method : 0.022256851196289062 length of segment : 182 time for calcul the mask position with numpy : 0.0017900466918945312 nb_pixel_total : 28624 time to create 1 rle with old method : 0.031722068786621094 length of segment : 523 time for calcul the mask position with numpy : 0.0025510787963867188 nb_pixel_total : 99503 time to create 1 rle with old method : 0.107208251953125 length of segment : 360 time for calcul the mask position with numpy : 0.0002579689025878906 nb_pixel_total : 4400 time to create 1 rle with old method : 0.004872798919677734 length of segment : 101 time for calcul the mask position with numpy : 0.004159450531005859 nb_pixel_total : 168874 time to create 1 rle with new method : 0.006032705307006836 length of segment : 581 time for calcul the mask position with numpy : 0.03818655014038086 nb_pixel_total : 1465785 time to create 1 rle with new method : 0.08020544052124023 length of segment : 2323 time for calcul the mask position with numpy : 0.026729106903076172 nb_pixel_total : 634562 time to create 1 rle with new method : 0.04331016540527344 length of segment : 2175 time for calcul the mask position with numpy : 0.002773284912109375 nb_pixel_total : 137091 time to create 1 rle with old method : 0.14701271057128906 length of segment : 442 time for calcul the mask position with numpy : 0.002862691879272461 nb_pixel_total : 120890 time to create 1 rle with old method : 0.13041234016418457 length of segment : 830 time for calcul the mask position with numpy : 0.0004456043243408203 nb_pixel_total : 29162 time to create 1 rle with old method : 0.03244304656982422 length of segment : 186 time for calcul the mask position with numpy : 0.00399017333984375 nb_pixel_total : 157733 time to create 1 rle with new method : 0.006464242935180664 length of segment : 449 time for calcul the mask position with numpy : 0.0010573863983154297 nb_pixel_total : 51279 time to create 1 rle with old method : 0.054135799407958984 length of segment : 340 time for calcul the mask position with numpy : 0.004975557327270508 nb_pixel_total : 201211 time to create 1 rle with new method : 0.014315366744995117 length of segment : 719 time for calcul the mask position with numpy : 0.004534721374511719 nb_pixel_total : 189718 time to create 1 rle with new method : 0.009572982788085938 length of segment : 470 time for calcul the mask position with numpy : 0.007862091064453125 nb_pixel_total : 452172 time to create 1 rle with new method : 0.018431663513183594 length of segment : 1191 time for calcul the mask position with numpy : 0.008075237274169922 nb_pixel_total : 207418 time to create 1 rle with new method : 0.009412288665771484 length of segment : 807 time for calcul the mask position with numpy : 0.0005650520324707031 nb_pixel_total : 8602 time to create 1 rle with old method : 0.010001897811889648 length of segment : 185 time for calcul the mask position with numpy : 0.0006546974182128906 nb_pixel_total : 14502 time to create 1 rle with old method : 0.01623368263244629 length of segment : 151 time for calcul the mask position with numpy : 0.0013148784637451172 nb_pixel_total : 29600 time to create 1 rle with old method : 0.03296709060668945 length of segment : 326 time for calcul the mask position with numpy : 0.0015895366668701172 nb_pixel_total : 58272 time to create 1 rle with old method : 0.06285929679870605 length of segment : 607 time for calcul the mask position with numpy : 0.00025773048400878906 nb_pixel_total : 3758 time to create 1 rle with old method : 0.004331827163696289 length of segment : 111 time for calcul the mask position with numpy : 0.003272533416748047 nb_pixel_total : 153615 time to create 1 rle with new method : 0.004582881927490234 length of segment : 756 time for calcul the mask position with numpy : 0.0003898143768310547 nb_pixel_total : 15759 time to create 1 rle with old method : 0.01750040054321289 length of segment : 178 time for calcul the mask position with numpy : 0.005231618881225586 nb_pixel_total : 229186 time to create 1 rle with new method : 0.007128477096557617 length of segment : 593 time for calcul the mask position with numpy : 0.00036835670471191406 nb_pixel_total : 6660 time to create 1 rle with old method : 0.007576942443847656 length of segment : 100 time for calcul the mask position with numpy : 0.00046634674072265625 nb_pixel_total : 8689 time to create 1 rle with old method : 0.010403156280517578 length of segment : 250 time for calcul the mask position with numpy : 0.0024073123931884766 nb_pixel_total : 98196 time to create 1 rle with old method : 0.10609626770019531 length of segment : 503 time for calcul the mask position with numpy : 0.0017533302307128906 nb_pixel_total : 82389 time to create 1 rle with old method : 0.08873748779296875 length of segment : 360 time for calcul the mask position with numpy : 0.00046753883361816406 nb_pixel_total : 12236 time to create 1 rle with old method : 0.013498783111572266 length of segment : 155 time for calcul the mask position with numpy : 0.0009944438934326172 nb_pixel_total : 18315 time to create 1 rle with old method : 0.02103877067565918 length of segment : 349 time for calcul the mask position with numpy : 0.0038614273071289062 nb_pixel_total : 173337 time to create 1 rle with new method : 0.00633549690246582 length of segment : 523 time for calcul the mask position with numpy : 0.0026781558990478516 nb_pixel_total : 120583 time to create 1 rle with old method : 0.13263416290283203 length of segment : 621 time for calcul the mask position with numpy : 0.001178741455078125 nb_pixel_total : 21861 time to create 1 rle with old method : 0.02486419677734375 length of segment : 386 time for calcul the mask position with numpy : 0.0027892589569091797 nb_pixel_total : 111403 time to create 1 rle with old method : 0.12729954719543457 length of segment : 715 time for calcul the mask position with numpy : 0.0010802745819091797 nb_pixel_total : 50390 time to create 1 rle with old method : 0.06153988838195801 length of segment : 209 time for calcul the mask position with numpy : 0.002948284149169922 nb_pixel_total : 153560 time to create 1 rle with new method : 0.005277872085571289 length of segment : 598 time for calcul the mask position with numpy : 0.0026428699493408203 nb_pixel_total : 89524 time to create 1 rle with old method : 0.10599398612976074 length of segment : 458 time for calcul the mask position with numpy : 0.0034770965576171875 nb_pixel_total : 146623 time to create 1 rle with old method : 0.1592707633972168 length of segment : 503 time for calcul the mask position with numpy : 0.0006024837493896484 nb_pixel_total : 19176 time to create 1 rle with old method : 0.023146390914916992 length of segment : 126 time for calcul the mask position with numpy : 0.0005972385406494141 nb_pixel_total : 13450 time to create 1 rle with old method : 0.016078472137451172 length of segment : 167 time for calcul the mask position with numpy : 0.003665447235107422 nb_pixel_total : 162147 time to create 1 rle with new method : 0.005187034606933594 length of segment : 562 time for calcul the mask position with numpy : 0.0025043487548828125 nb_pixel_total : 48881 time to create 1 rle with old method : 0.05367612838745117 length of segment : 321 time for calcul the mask position with numpy : 0.00045108795166015625 nb_pixel_total : 5304 time to create 1 rle with old method : 0.0062558650970458984 length of segment : 154 time for calcul the mask position with numpy : 0.006967782974243164 nb_pixel_total : 231109 time to create 1 rle with new method : 0.012712717056274414 length of segment : 662 time for calcul the mask position with numpy : 0.002235889434814453 nb_pixel_total : 85478 time to create 1 rle with old method : 0.08954429626464844 length of segment : 435 time for calcul the mask position with numpy : 0.0035827159881591797 nb_pixel_total : 229183 time to create 1 rle with new method : 0.00838160514831543 length of segment : 664 time for calcul the mask position with numpy : 0.0016613006591796875 nb_pixel_total : 78591 time to create 1 rle with old method : 0.08662748336791992 length of segment : 462 time for calcul the mask position with numpy : 0.001697540283203125 nb_pixel_total : 31869 time to create 1 rle with old method : 0.03920102119445801 length of segment : 349 time for calcul the mask position with numpy : 0.0001513957977294922 nb_pixel_total : 5811 time to create 1 rle with old method : 0.006619453430175781 length of segment : 155 time for calcul the mask position with numpy : 0.0010373592376708984 nb_pixel_total : 54379 time to create 1 rle with old method : 0.05963921546936035 length of segment : 284 time for calcul the mask position with numpy : 0.00408935546875 nb_pixel_total : 172351 time to create 1 rle with new method : 0.0060117244720458984 length of segment : 597 time for calcul the mask position with numpy : 0.0008640289306640625 nb_pixel_total : 23877 time to create 1 rle with old method : 0.027363300323486328 length of segment : 182 time for calcul the mask position with numpy : 0.0009641647338867188 nb_pixel_total : 23819 time to create 1 rle with old method : 0.027216196060180664 length of segment : 208 time for calcul the mask position with numpy : 0.013579607009887695 nb_pixel_total : 662296 time to create 1 rle with new method : 0.0190582275390625 length of segment : 1503 time for calcul the mask position with numpy : 0.00018024444580078125 nb_pixel_total : 5416 time to create 1 rle with old method : 0.006378889083862305 length of segment : 121 time for calcul the mask position with numpy : 0.0020020008087158203 nb_pixel_total : 78791 time to create 1 rle with old method : 0.08893799781799316 length of segment : 312 time for calcul the mask position with numpy : 0.013930559158325195 nb_pixel_total : 368527 time to create 1 rle with new method : 0.02247786521911621 length of segment : 1582 time for calcul the mask position with numpy : 0.00029277801513671875 nb_pixel_total : 2648 time to create 1 rle with old method : 0.003258943557739258 length of segment : 90 time for calcul the mask position with numpy : 0.018594026565551758 nb_pixel_total : 739093 time to create 1 rle with new method : 0.043929338455200195 length of segment : 2325 time for calcul the mask position with numpy : 0.0010035037994384766 nb_pixel_total : 51615 time to create 1 rle with old method : 0.05611014366149902 length of segment : 339 time for calcul the mask position with numpy : 0.010999202728271484 nb_pixel_total : 444947 time to create 1 rle with new method : 0.018252134323120117 length of segment : 825 time for calcul the mask position with numpy : 0.0005462169647216797 nb_pixel_total : 12327 time to create 1 rle with old method : 0.020139217376708984 length of segment : 124 time for calcul the mask position with numpy : 0.0005218982696533203 nb_pixel_total : 19904 time to create 1 rle with old method : 0.022104978561401367 length of segment : 130 time for calcul the mask position with numpy : 0.0019440650939941406 nb_pixel_total : 68590 time to create 1 rle with old method : 0.1025240421295166 length of segment : 235 time for calcul the mask position with numpy : 0.0010352134704589844 nb_pixel_total : 52802 time to create 1 rle with old method : 0.05824685096740723 length of segment : 346 time for calcul the mask position with numpy : 0.004714488983154297 nb_pixel_total : 179582 time to create 1 rle with new method : 0.006567478179931641 length of segment : 618 time for calcul the mask position with numpy : 0.0005447864532470703 nb_pixel_total : 13506 time to create 1 rle with old method : 0.0156097412109375 length of segment : 131 time for calcul the mask position with numpy : 0.0007927417755126953 nb_pixel_total : 23315 time to create 1 rle with old method : 0.02781057357788086 length of segment : 201 time for calcul the mask position with numpy : 0.015774250030517578 nb_pixel_total : 672283 time to create 1 rle with new method : 0.022815704345703125 length of segment : 1567 time for calcul the mask position with numpy : 0.0018782615661621094 nb_pixel_total : 47638 time to create 1 rle with old method : 0.051622629165649414 length of segment : 295 time for calcul the mask position with numpy : 0.002381563186645508 nb_pixel_total : 71019 time to create 1 rle with old method : 0.10148239135742188 length of segment : 304 time for calcul the mask position with numpy : 0.0007450580596923828 nb_pixel_total : 10663 time to create 1 rle with old method : 0.012581110000610352 length of segment : 209 time for calcul the mask position with numpy : 0.0007617473602294922 nb_pixel_total : 31185 time to create 1 rle with old method : 0.03392767906188965 length of segment : 197 time for calcul the mask position with numpy : 0.018596649169921875 nb_pixel_total : 594878 time to create 1 rle with new method : 0.03064703941345215 length of segment : 855 time for calcul the mask position with numpy : 0.010048389434814453 nb_pixel_total : 260218 time to create 1 rle with new method : 0.015556097030639648 length of segment : 877 time for calcul the mask position with numpy : 0.002234220504760742 nb_pixel_total : 103163 time to create 1 rle with old method : 0.11079025268554688 length of segment : 305 time for calcul the mask position with numpy : 0.0005168914794921875 nb_pixel_total : 22978 time to create 1 rle with old method : 0.02505970001220703 length of segment : 246 time for calcul the mask position with numpy : 0.006533622741699219 nb_pixel_total : 246601 time to create 1 rle with new method : 0.020081281661987305 length of segment : 997 time for calcul the mask position with numpy : 0.0005474090576171875 nb_pixel_total : 18491 time to create 1 rle with old method : 0.02022695541381836 length of segment : 151 time for calcul the mask position with numpy : 0.007118940353393555 nb_pixel_total : 234233 time to create 1 rle with new method : 0.011461973190307617 length of segment : 1038 time spent for convertir_results : 18.979004621505737 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 137 chid ids of type : 3663 Number RLEs to save : 76396 save missing photos in datou_result : time spend for datou_step_exec : 94.86087822914124 time spend to save output : 4.219311237335205 total time spend for step 1 : 99.08018946647644 step2:crop_condition Wed Feb 12 12:26:11 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 137 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 17 About to insert : list_path_to_insert length 17 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 11 About to insert : list_path_to_insert length 11 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 34 About to insert : list_path_to_insert length 34 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1335238113, 1335237999, 1335237967, 1335237931, 1335237877, 1335237738, 1335237734, 1335237716, 1335237690, 1335237669] Looping around the photos to save general results len do output : 82 /-3671663078Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663095Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663112Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663108Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663104Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663106Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663125Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663119Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663123Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663139Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663142Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663173Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663185Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663191Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663187Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663200Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663202Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663089Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663085Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663090Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663162Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663172Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663179Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663178Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663176Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663194Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663190Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663211Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663088Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663094Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663148Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663147Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663145Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663086Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663082Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663080Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663079Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663087Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663092Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663097Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663100Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663111Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663105Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663110Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663117Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663124Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663126Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663137Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663136Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663121Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663128Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663143Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663161Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663158Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663169Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663170Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663168Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663180Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663184Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663182Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663198Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663197Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663186Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663209Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663207Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663210Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663205Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663103Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663107Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663120Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663129Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663159Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663157Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663174Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663206Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663093Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663081Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663144Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663175Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663181Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663199Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3671663135Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335238113', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237999', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237967', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237931', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237877', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237738', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237734', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237716', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237690', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237669', None, None, None, None, None, '2579320') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 256 time used for this insertion : 0.036226749420166016 save_final save missing photos in datou_result : time spend for datou_step_exec : 47.62718653678894 time spend to save output : 0.03878641128540039 total time spend for step 2 : 47.66597294807434 step3:thcl Wed Feb 12 12:26:59 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.003482818603515625 time to convert the images to numpy array : 0.003966331481933594 time to import caffe and check if the image exist : 0.012633085250854492 time to convert the images to numpy array : 0.13434529304504395 time to import caffe and check if the image exist : 0.011896371841430664 time to convert the images to numpy array : 0.13812041282653809 time to import caffe and check if the image exist : 0.018561124801635742 time to convert the images to numpy array : 0.5937292575836182 time to import caffe and check if the image exist : 0.01180124282836914 time to convert the images to numpy array : 0.6188068389892578 time to import caffe and check if the image exist : 0.00823211669921875 time to convert the images to numpy array : 0.6358907222747803 time to import caffe and check if the image exist : 0.012949943542480469 time to convert the images to numpy array : 0.6359467506408691 time to import caffe and check if the image exist : 0.00999760627746582 time to convert the images to numpy array : 0.6394598484039307 time to import caffe and check if the image exist : 0.010303020477294922 time to convert the images to numpy array : 0.6529881954193115 time to import caffe and check if the image exist : 0.009976387023925781 time to convert the images to numpy array : 0.6585321426391602 total time to convert the images to numpy array : 0.8772945404052734 list photo_ids error: [] list photo_ids correct : [-3671663135, -3671663209, -3671663207, -3671663210, -3671663205, -3671663103, -3671663107, -3671663120, -3671663129, -3671663159, -3671663157, -3671663174, -3671663206, -3671663093, -3671663081, -3671663144, -3671663175, -3671663181, -3671663199, -3671663169, -3671663170, -3671663168, -3671663180, -3671663184, -3671663182, -3671663198, -3671663197, -3671663186, -3671663085, -3671663090, -3671663162, -3671663172, -3671663179, -3671663178, -3671663176, -3671663194, -3671663190, -3671663139, -3671663142, -3671663173, -3671663185, -3671663191, -3671663187, -3671663200, -3671663202, -3671663089, -3671663211, -3671663088, -3671663094, -3671663148, -3671663147, -3671663145, -3671663086, -3671663082, -3671663080, -3671663079, -3671663087, -3671663092, -3671663097, -3671663100, -3671663111, -3671663105, -3671663110, -3671663117, -3671663124, -3671663126, -3671663137, -3671663136, -3671663121, -3671663128, -3671663143, -3671663161, -3671663158, -3671663078, -3671663095, -3671663112, -3671663108, -3671663104, -3671663106, -3671663125, -3671663119, -3671663123] number of photos to traite : 82 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 2149 wait 20 seconds l 3637 free memory gpu now : 2149 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 1930 wait 20 seconds l 3637 free memory gpu now : 1930 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.2005290985107422 time used to do the prediction : 0.20924615859985352 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 1928 wait 20 seconds l 3637 free memory gpu now : 1928 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 3121 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.1544101238250732 time used to do the prediction : 0.2660043239593506 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 1.9550323486328125e-05 save missing photos in datou_result : time spend for datou_step_exec : 70.99995255470276 time spend to save output : 0.0006418228149414062 total time spend for step 3 : 71.0005943775177 step4:merge_mask_thcl_custom Wed Feb 12 12:28:10 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 137 chid ids of type : 3663 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1335238113, 1335237999, 1335237967, 1335237931, 1335237877, 1335237738, 1335237734, 1335237716, 1335237690, 1335237669] Looping around the photos to save general results len do output : 10 /1335238113Didn't retrieve data .Didn't retrieve data . /1335237999Didn't retrieve data .Didn't retrieve data . /1335237967Didn't retrieve data .Didn't retrieve data . /1335237931Didn't retrieve data .Didn't retrieve data . /1335237877Didn't retrieve data .Didn't retrieve data . /1335237738Didn't retrieve data .Didn't retrieve data . /1335237734Didn't retrieve data .Didn't retrieve data . /1335237716Didn't retrieve data .Didn't retrieve data . /1335237690Didn't retrieve data .Didn't retrieve data . /1335237669Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335238113', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237999', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237967', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237931', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237877', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237738', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237734', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237716', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237690', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237669', None, None, None, None, None, '2579320') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.5321991443634033 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.06780385971069336 time spend to save output : 0.5327181816101074 total time spend for step 4 : 0.6005220413208008 step5:rle_unique_nms_with_priority Wed Feb 12 12:28:10 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 16 nb_hashtags : 7 time to prepare the origin masks : 18.92142939567566 time for calcul the mask position with numpy : 0.8449521064758301 nb_pixel_total : 5348691 time to create 1 rle with new method : 1.0674641132354736 time for calcul the mask position with numpy : 0.030621051788330078 nb_pixel_total : 36315 time to create 1 rle with old method : 0.039138078689575195 time for calcul the mask position with numpy : 0.03046441078186035 nb_pixel_total : 24232 time to create 1 rle with old method : 0.02729511260986328 time for calcul the mask position with numpy : 0.03156852722167969 nb_pixel_total : 175763 time to create 1 rle with new method : 1.4085240364074707 time for calcul the mask position with numpy : 0.030369281768798828 nb_pixel_total : 93887 time to create 1 rle with old method : 0.11144709587097168 time for calcul the mask position with numpy : 0.02942037582397461 nb_pixel_total : 39277 time to create 1 rle with old method : 0.04213285446166992 time for calcul the mask position with numpy : 0.02687239646911621 nb_pixel_total : 178702 time to create 1 rle with new method : 0.8219194412231445 time for calcul the mask position with numpy : 0.03310537338256836 nb_pixel_total : 1561 time to create 1 rle with old method : 0.0025060176849365234 time for calcul the mask position with numpy : 0.05013132095336914 nb_pixel_total : 1504239 time to create 1 rle with new method : 0.8962876796722412 time for calcul the mask position with numpy : 0.025430917739868164 nb_pixel_total : 189638 time to create 1 rle with new method : 0.5220546722412109 time for calcul the mask position with numpy : 0.028183460235595703 nb_pixel_total : 145734 time to create 1 rle with old method : 0.1646888256072998 time for calcul the mask position with numpy : 0.04410362243652344 nb_pixel_total : 117888 time to create 1 rle with old method : 0.13551855087280273 time for calcul the mask position with numpy : 0.042440176010131836 nb_pixel_total : 125880 time to create 1 rle with old method : 0.13496971130371094 time for calcul the mask position with numpy : 0.0479278564453125 nb_pixel_total : 138929 time to create 1 rle with old method : 0.16421055793762207 time for calcul the mask position with numpy : 0.04656505584716797 nb_pixel_total : 33145 time to create 1 rle with old method : 0.05359196662902832 time for calcul the mask position with numpy : 0.04497337341308594 nb_pixel_total : 21897 time to create 1 rle with old method : 0.02737140655517578 time for calcul the mask position with numpy : 0.045484066009521484 nb_pixel_total : 118622 time to create 1 rle with old method : 0.13909268379211426 create new chi : 7.350031614303589 time to delete rle : 0.013455867767333984 batch 1 Loaded 17 chid ids of type : 3726 Number RLEs to save : 18609 TO DO : save crop sub photo not yet done ! save time : 1.3109793663024902 nb_obj : 7 nb_hashtags : 3 time to prepare the origin masks : 9.769436836242676 time for calcul the mask position with numpy : 0.6437158584594727 nb_pixel_total : 3380518 time to create 1 rle with new method : 1.7460968494415283 time for calcul the mask position with numpy : 0.07061576843261719 nb_pixel_total : 788285 time to create 1 rle with new method : 1.0587961673736572 time for calcul the mask position with numpy : 0.051320552825927734 nb_pixel_total : 177262 time to create 1 rle with new method : 1.165266990661621 time for calcul the mask position with numpy : 0.0470728874206543 nb_pixel_total : 220714 time to create 1 rle with new method : 0.7372779846191406 time for calcul the mask position with numpy : 0.053043365478515625 nb_pixel_total : 129535 time to create 1 rle with old method : 0.14496493339538574 time for calcul the mask position with numpy : 0.20970821380615234 nb_pixel_total : 3497941 time to create 1 rle with new method : 0.7043650150299072 time for calcul the mask position with numpy : 0.0412139892578125 nb_pixel_total : 54798 time to create 1 rle with old method : 0.06081724166870117 time for calcul the mask position with numpy : 0.041393280029296875 nb_pixel_total : 45347 time to create 1 rle with old method : 0.04865264892578125 create new chi : 7.021751165390015 time to delete rle : 0.0016713142395019531 batch 1 Loaded 8 chid ids of type : 3726 Number RLEs to save : 14747 TO DO : save crop sub photo not yet done ! save time : 0.9239470958709717 nb_obj : 14 nb_hashtags : 6 time to prepare the origin masks : 16.029175996780396 time for calcul the mask position with numpy : 0.2945876121520996 nb_pixel_total : 5927643 time to create 1 rle with new method : 1.3422245979309082 time for calcul the mask position with numpy : 0.025209665298461914 nb_pixel_total : 7265 time to create 1 rle with old method : 0.008259773254394531 time for calcul the mask position with numpy : 0.03026747703552246 nb_pixel_total : 314673 time to create 1 rle with new method : 0.8936557769775391 time for calcul the mask position with numpy : 0.025211811065673828 nb_pixel_total : 68254 time to create 1 rle with old method : 0.07299113273620605 time for calcul the mask position with numpy : 0.025484085083007812 nb_pixel_total : 94977 time to create 1 rle with old method : 0.10238075256347656 time for calcul the mask position with numpy : 0.024273395538330078 nb_pixel_total : 18960 time to create 1 rle with old method : 0.020964860916137695 time for calcul the mask position with numpy : 0.025990009307861328 nb_pixel_total : 149940 time to create 1 rle with old method : 0.16297364234924316 time for calcul the mask position with numpy : 0.02528858184814453 nb_pixel_total : 73698 time to create 1 rle with old method : 0.08568501472473145 time for calcul the mask position with numpy : 0.04370403289794922 nb_pixel_total : 587778 time to create 1 rle with new method : 1.88228178024292 time for calcul the mask position with numpy : 0.03905344009399414 nb_pixel_total : 15339 time to create 1 rle with old method : 0.016314268112182617 time for calcul the mask position with numpy : 0.03890061378479004 nb_pixel_total : 11730 time to create 1 rle with old method : 0.012211322784423828 time for calcul the mask position with numpy : 0.03981280326843262 nb_pixel_total : 170983 time to create 1 rle with new method : 0.53863525390625 time for calcul the mask position with numpy : 0.03864240646362305 nb_pixel_total : 36749 time to create 1 rle with old method : 0.03924679756164551 time for calcul the mask position with numpy : 0.04729056358337402 nb_pixel_total : 757046 time to create 1 rle with new method : 0.9214742183685303 time for calcul the mask position with numpy : 0.04175233840942383 nb_pixel_total : 59365 time to create 1 rle with old method : 0.06525206565856934 create new chi : 7.094217300415039 time to delete rle : 0.0020198822021484375 batch 1 Loaded 15 chid ids of type : 3726 Number RLEs to save : 18199 TO DO : save crop sub photo not yet done ! save time : 1.2360949516296387 nb_obj : 17 nb_hashtags : 7 time to prepare the origin masks : 20.815954446792603 time for calcul the mask position with numpy : 0.7225620746612549 nb_pixel_total : 6772281 time to create 1 rle with new method : 0.6678588390350342 time for calcul the mask position with numpy : 0.02807903289794922 nb_pixel_total : 15699 time to create 1 rle with old method : 0.0170438289642334 time for calcul the mask position with numpy : 0.04303407669067383 nb_pixel_total : 21609 time to create 1 rle with old method : 0.023280858993530273 time for calcul the mask position with numpy : 0.04258227348327637 nb_pixel_total : 19067 time to create 1 rle with old method : 0.020947694778442383 time for calcul the mask position with numpy : 0.046248674392700195 nb_pixel_total : 197204 time to create 1 rle with new method : 1.5302391052246094 time for calcul the mask position with numpy : 0.041535139083862305 nb_pixel_total : 62304 time to create 1 rle with old method : 0.06821417808532715 time for calcul the mask position with numpy : 0.043994903564453125 nb_pixel_total : 99269 time to create 1 rle with old method : 0.10979485511779785 time for calcul the mask position with numpy : 0.04139399528503418 nb_pixel_total : 2064 time to create 1 rle with old method : 0.0024156570434570312 time for calcul the mask position with numpy : 0.041556596755981445 nb_pixel_total : 22032 time to create 1 rle with old method : 0.024614572525024414 time for calcul the mask position with numpy : 0.04402613639831543 nb_pixel_total : 163354 time to create 1 rle with new method : 0.9635219573974609 time for calcul the mask position with numpy : 0.04222464561462402 nb_pixel_total : 654090 time to create 1 rle with new method : 1.0798931121826172 time for calcul the mask position with numpy : 0.042183876037597656 nb_pixel_total : 2278 time to create 1 rle with old method : 0.0028111934661865234 time for calcul the mask position with numpy : 0.03671097755432129 nb_pixel_total : 1792 time to create 1 rle with old method : 0.0022165775299072266 time for calcul the mask position with numpy : 0.0341947078704834 nb_pixel_total : 48026 time to create 1 rle with old method : 0.05169534683227539 time for calcul the mask position with numpy : 0.041230201721191406 nb_pixel_total : 172938 time to create 1 rle with new method : 1.5041050910949707 time for calcul the mask position with numpy : 0.03553128242492676 nb_pixel_total : 2134 time to create 1 rle with old method : 0.002499103546142578 time for calcul the mask position with numpy : 0.027150630950927734 nb_pixel_total : 9782 time to create 1 rle with old method : 0.011284351348876953 time for calcul the mask position with numpy : 0.02644062042236328 nb_pixel_total : 28477 time to create 1 rle with old method : 0.031166553497314453 create new chi : 7.671862840652466 time to delete rle : 0.0017774105072021484 batch 1 Loaded 18 chid ids of type : 3726 Number RLEs to save : 13733 TO DO : save crop sub photo not yet done ! save time : 1.0800206661224365 nb_obj : 11 nb_hashtags : 4 time to prepare the origin masks : 15.220281600952148 time for calcul the mask position with numpy : 1.128816843032837 nb_pixel_total : 5199024 time to create 1 rle with new method : 0.7943630218505859 time for calcul the mask position with numpy : 0.05326533317565918 nb_pixel_total : 157312 time to create 1 rle with new method : 1.5196304321289062 time for calcul the mask position with numpy : 0.04674673080444336 nb_pixel_total : 226087 time to create 1 rle with new method : 1.8141229152679443 time for calcul the mask position with numpy : 0.04821896553039551 nb_pixel_total : 57196 time to create 1 rle with old method : 0.06328654289245605 time for calcul the mask position with numpy : 0.05410170555114746 nb_pixel_total : 633739 time to create 1 rle with new method : 0.7612521648406982 time for calcul the mask position with numpy : 0.06931352615356445 nb_pixel_total : 1474988 time to create 1 rle with new method : 1.295783519744873 time for calcul the mask position with numpy : 0.04521298408508301 nb_pixel_total : 65586 time to create 1 rle with old method : 0.06914305686950684 time for calcul the mask position with numpy : 0.044072866439819336 nb_pixel_total : 4386 time to create 1 rle with old method : 0.004815101623535156 time for calcul the mask position with numpy : 0.05218315124511719 nb_pixel_total : 136939 time to create 1 rle with old method : 0.14495062828063965 time for calcul the mask position with numpy : 0.04994082450866699 nb_pixel_total : 120573 time to create 1 rle with old method : 0.15384697914123535 time for calcul the mask position with numpy : 0.04314255714416504 nb_pixel_total : 189572 time to create 1 rle with new method : 1.76247239112854 time for calcul the mask position with numpy : 0.039691925048828125 nb_pixel_total : 28998 time to create 1 rle with old method : 0.03081512451171875 create new chi : 10.31813097000122 time to delete rle : 0.0026085376739501953 batch 1 Loaded 12 chid ids of type : 3726 Number RLEs to save : 19052 TO DO : save crop sub photo not yet done ! save time : 9.574930429458618 nb_obj : 12 nb_hashtags : 5 time to prepare the origin masks : 21.758148431777954 time for calcul the mask position with numpy : 1.7381479740142822 nb_pixel_total : 7586462 time to create 1 rle with new method : 1.4240801334381104 time for calcul the mask position with numpy : 0.04358839988708496 nb_pixel_total : 82158 time to create 1 rle with old method : 0.08804583549499512 time for calcul the mask position with numpy : 0.04336261749267578 nb_pixel_total : 15701 time to create 1 rle with old method : 0.025322914123535156 time for calcul the mask position with numpy : 0.04055023193359375 nb_pixel_total : 8660 time to create 1 rle with old method : 0.00925135612487793 time for calcul the mask position with numpy : 0.03961825370788574 nb_pixel_total : 98058 time to create 1 rle with old method : 0.10223078727722168 time for calcul the mask position with numpy : 0.04110455513000488 nb_pixel_total : 228817 time to create 1 rle with new method : 0.9981482028961182 time for calcul the mask position with numpy : 0.045456886291503906 nb_pixel_total : 153576 time to create 1 rle with new method : 0.7443509101867676 time for calcul the mask position with numpy : 0.04397153854370117 nb_pixel_total : 3739 time to create 1 rle with old method : 0.0044596195220947266 time for calcul the mask position with numpy : 0.04641222953796387 nb_pixel_total : 58129 time to create 1 rle with old method : 0.06854534149169922 time for calcul the mask position with numpy : 0.04309558868408203 nb_pixel_total : 29576 time to create 1 rle with old method : 0.03284716606140137 time for calcul the mask position with numpy : 0.045180320739746094 nb_pixel_total : 14379 time to create 1 rle with old method : 0.015527725219726562 time for calcul the mask position with numpy : 0.04567670822143555 nb_pixel_total : 8561 time to create 1 rle with old method : 0.00927424430847168 time for calcul the mask position with numpy : 0.02579784393310547 nb_pixel_total : 6584 time to create 1 rle with old method : 0.007592201232910156 create new chi : 5.883432865142822 time to delete rle : 0.0011715888977050781 batch 1 Loaded 13 chid ids of type : 3726 Number RLEs to save : 10416 TO DO : save crop sub photo not yet done ! save time : 1.51202392578125 nb_obj : 5 nb_hashtags : 3 time to prepare the origin masks : 8.07936429977417 time for calcul the mask position with numpy : 0.6470131874084473 nb_pixel_total : 7835940 time to create 1 rle with new method : 1.0513834953308105 time for calcul the mask position with numpy : 0.04471397399902344 nb_pixel_total : 50067 time to create 1 rle with old method : 0.05572199821472168 time for calcul the mask position with numpy : 0.04167532920837402 nb_pixel_total : 155043 time to create 1 rle with new method : 1.2426176071166992 time for calcul the mask position with numpy : 0.04876899719238281 nb_pixel_total : 21823 time to create 1 rle with old method : 0.0238037109375 time for calcul the mask position with numpy : 0.055614471435546875 nb_pixel_total : 111048 time to create 1 rle with old method : 0.1242063045501709 time for calcul the mask position with numpy : 0.05171036720275879 nb_pixel_total : 120479 time to create 1 rle with old method : 0.13640475273132324 create new chi : 3.598484754562378 time to delete rle : 0.0008311271667480469 batch 1 Loaded 6 chid ids of type : 3726 Number RLEs to save : 7100 TO DO : save crop sub photo not yet done ! save time : 0.4269397258758545 nb_obj : 14 nb_hashtags : 8 time to prepare the origin masks : 14.564289569854736 time for calcul the mask position with numpy : 0.5173919200897217 nb_pixel_total : 7396289 time to create 1 rle with new method : 1.2557439804077148 time for calcul the mask position with numpy : 0.031638383865356445 nb_pixel_total : 6336 time to create 1 rle with old method : 0.00774073600769043 time for calcul the mask position with numpy : 0.031070232391357422 nb_pixel_total : 31743 time to create 1 rle with old method : 0.03485822677612305 time for calcul the mask position with numpy : 0.03332948684692383 nb_pixel_total : 229025 time to create 1 rle with new method : 0.39504551887512207 time for calcul the mask position with numpy : 0.03236269950866699 nb_pixel_total : 48786 time to create 1 rle with old method : 0.056720733642578125 time for calcul the mask position with numpy : 0.029649734497070312 nb_pixel_total : 54331 time to create 1 rle with old method : 0.05834674835205078 time for calcul the mask position with numpy : 0.030490875244140625 nb_pixel_total : 162285 time to create 1 rle with new method : 1.55615234375 time for calcul the mask position with numpy : 0.029036521911621094 nb_pixel_total : 5792 time to create 1 rle with old method : 0.0067021846771240234 time for calcul the mask position with numpy : 0.03072071075439453 nb_pixel_total : 7898 time to create 1 rle with old method : 0.008985280990600586 time for calcul the mask position with numpy : 0.03081655502319336 nb_pixel_total : 146175 time to create 1 rle with old method : 0.1549973487854004 time for calcul the mask position with numpy : 0.04952859878540039 nb_pixel_total : 19074 time to create 1 rle with old method : 0.019605398178100586 time for calcul the mask position with numpy : 0.05099368095397949 nb_pixel_total : 5298 time to create 1 rle with old method : 0.0057332515716552734 time for calcul the mask position with numpy : 0.05740213394165039 nb_pixel_total : 78823 time to create 1 rle with old method : 0.08701467514038086 time for calcul the mask position with numpy : 0.045773983001708984 nb_pixel_total : 13364 time to create 1 rle with old method : 0.014268159866333008 time for calcul the mask position with numpy : 0.04224729537963867 nb_pixel_total : 89181 time to create 1 rle with old method : 0.09638357162475586 create new chi : 4.921849250793457 time to delete rle : 0.0012083053588867188 batch 1 Loaded 15 chid ids of type : 3726 Number RLEs to save : 12032 TO DO : save crop sub photo not yet done ! save time : 0.6948292255401611 nb_obj : 15 nb_hashtags : 6 time to prepare the origin masks : 12.683538675308228 time for calcul the mask position with numpy : 0.6015856266021729 nb_pixel_total : 5941926 time to create 1 rle with new method : 0.7737796306610107 time for calcul the mask position with numpy : 0.04528927803039551 nb_pixel_total : 68348 time to create 1 rle with old method : 0.07389187812805176 time for calcul the mask position with numpy : 0.04407310485839844 nb_pixel_total : 23807 time to create 1 rle with old method : 0.026227235794067383 time for calcul the mask position with numpy : 0.04050111770629883 nb_pixel_total : 956 time to create 1 rle with old method : 0.0012540817260742188 time for calcul the mask position with numpy : 0.026422739028930664 nb_pixel_total : 12280 time to create 1 rle with old method : 0.0138702392578125 time for calcul the mask position with numpy : 0.02952885627746582 nb_pixel_total : 444781 time to create 1 rle with new method : 0.5027272701263428 time for calcul the mask position with numpy : 0.028324127197265625 nb_pixel_total : 52062 time to create 1 rle with old method : 0.05661320686340332 time for calcul the mask position with numpy : 0.026833772659301758 nb_pixel_total : 78675 time to create 1 rle with old method : 0.08469581604003906 time for calcul the mask position with numpy : 0.030051231384277344 nb_pixel_total : 467258 time to create 1 rle with new method : 0.7201142311096191 time for calcul the mask position with numpy : 0.026576995849609375 nb_pixel_total : 19816 time to create 1 rle with old method : 0.023508548736572266 time for calcul the mask position with numpy : 0.025000572204589844 nb_pixel_total : 2633 time to create 1 rle with old method : 0.0033674240112304688 time for calcul the mask position with numpy : 0.02550196647644043 nb_pixel_total : 123536 time to create 1 rle with old method : 0.15146994590759277 time for calcul the mask position with numpy : 0.0249330997467041 nb_pixel_total : 5382 time to create 1 rle with old method : 0.005925178527832031 time for calcul the mask position with numpy : 0.03474140167236328 nb_pixel_total : 661087 time to create 1 rle with new method : 0.5609111785888672 time for calcul the mask position with numpy : 0.03153204917907715 nb_pixel_total : 368126 time to create 1 rle with new method : 0.6345052719116211 time for calcul the mask position with numpy : 0.024204730987548828 nb_pixel_total : 23727 time to create 1 rle with old method : 0.02425837516784668 create new chi : 4.88097620010376 time to delete rle : 0.0014646053314208984 batch 1 Loaded 16 chid ids of type : 3726 Number RLEs to save : 17977 TO DO : save crop sub photo not yet done ! save time : 1.0640082359313965 nb_obj : 12 nb_hashtags : 5 time to prepare the origin masks : 10.801612615585327 time for calcul the mask position with numpy : 0.39059925079345703 nb_pixel_total : 6791951 time to create 1 rle with new method : 0.4708678722381592 time for calcul the mask position with numpy : 0.026270389556884766 nb_pixel_total : 259733 time to create 1 rle with new method : 0.6831045150756836 time for calcul the mask position with numpy : 0.029043197631835938 nb_pixel_total : 114104 time to create 1 rle with old method : 0.1274397373199463 time for calcul the mask position with numpy : 0.028071880340576172 nb_pixel_total : 57493 time to create 1 rle with old method : 0.06429123878479004 time for calcul the mask position with numpy : 0.025278091430664062 nb_pixel_total : 70784 time to create 1 rle with old method : 0.07921242713928223 time for calcul the mask position with numpy : 0.025871753692626953 nb_pixel_total : 22965 time to create 1 rle with old method : 0.025196313858032227 time for calcul the mask position with numpy : 0.026305198669433594 nb_pixel_total : 10574 time to create 1 rle with old method : 0.01204061508178711 time for calcul the mask position with numpy : 0.028412342071533203 nb_pixel_total : 47416 time to create 1 rle with old method : 0.051009178161621094 time for calcul the mask position with numpy : 0.03022146224975586 nb_pixel_total : 672075 time to create 1 rle with new method : 0.616091251373291 time for calcul the mask position with numpy : 0.026099443435668945 nb_pixel_total : 179535 time to create 1 rle with new method : 0.8307821750640869 time for calcul the mask position with numpy : 0.030788183212280273 nb_pixel_total : 31071 time to create 1 rle with old method : 0.03235983848571777 time for calcul the mask position with numpy : 0.027992963790893555 nb_pixel_total : 13439 time to create 1 rle with old method : 0.014747142791748047 time for calcul the mask position with numpy : 0.03223133087158203 nb_pixel_total : 23260 time to create 1 rle with old method : 0.02414846420288086 create new chi : 3.8982322216033936 time to delete rle : 0.0012962818145751953 batch 1 Loaded 13 chid ids of type : 3726 Number RLEs to save : 13064 TO DO : save crop sub photo not yet done ! save time : 1.2262051105499268 map_output_result : {1335238113: (0.0, 'Should be the crop_list due to order', 0.0), 1335237999: (0.0, 'Should be the crop_list due to order', 0.0), 1335237967: (0.0, 'Should be the crop_list due to order', 0.0), 1335237931: (0.0, 'Should be the crop_list due to order', 0.0), 1335237877: (0.0, 'Should be the crop_list due to order', 0.0), 1335237738: (0.0, 'Should be the crop_list due to order', 0.0), 1335237734: (0.0, 'Should be the crop_list due to order', 0.0), 1335237716: (0.0, 'Should be the crop_list due to order', 0.0), 1335237690: (0.0, 'Should be the crop_list due to order', 0.0), 1335237669: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1335238113, 1335237999, 1335237967, 1335237931, 1335237877, 1335237738, 1335237734, 1335237716, 1335237690, 1335237669] Looping around the photos to save general results len do output : 10 /1335238113.Didn't retrieve data . /1335237999.Didn't retrieve data . /1335237967.Didn't retrieve data . /1335237931.Didn't retrieve data . /1335237877.Didn't retrieve data . /1335237738.Didn't retrieve data . /1335237734.Didn't retrieve data . /1335237716.Didn't retrieve data . /1335237690.Didn't retrieve data . /1335237669.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335238113', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237999', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237967', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237931', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237877', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237738', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237734', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237716', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237690', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237669', None, None, None, None, None, '2579320') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.07238626480102539 save_final save missing photos in datou_result : time spend for datou_step_exec : 234.28837084770203 time spend to save output : 0.07613730430603027 total time spend for step 5 : 234.36450815200806 step6:crop_condition Wed Feb 12 12:32:04 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 133 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! About to upload 8 photos upload in portfolio : 4869462 init cache_photo without model_param we have 8 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739359931_3348977 we have uploaded 8 photos in the portfolio 4869462 time of upload the photos Elapsed time : 3.378714084625244 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! About to upload 9 photos upload in portfolio : 4869462 init cache_photo without model_param we have 9 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739359940_3348977 we have uploaded 9 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.5639123916625977 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 6 About to insert : list_path_to_insert length 6 new photo from crops ! About to upload 6 photos upload in portfolio : 4869462 init cache_photo without model_param we have 6 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739359947_3348977 we have uploaded 6 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.4681684970855713 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739359951_3348977 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.7436764240264893 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! About to upload 5 photos upload in portfolio : 4869462 init cache_photo without model_param we have 5 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739359954_3348977 we have uploaded 5 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.266505479812622 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1739359959_3348977 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.299177408218384 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1335238113, 1335237999, 1335237967, 1335237931, 1335237877, 1335237738, 1335237734, 1335237716, 1335237690, 1335237669] Looping around the photos to save general results len do output : 32 /1337084590Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084593Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084597Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084600Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084603Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084604Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084605Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084606Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084859Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084863Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084865Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084866Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084867Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084868Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084869Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084870Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084871Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084875Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084876Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084877Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084878Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084879Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084880Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084882Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084883Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084909Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084910Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084911Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084912Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084913Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084995Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1337084998Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335238113', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237999', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237967', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237931', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237877', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237738', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237734', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237716', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237690', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237669', None, None, None, None, None, '2579320') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 106 time used for this insertion : 0.041626691818237305 save_final save missing photos in datou_result : time spend for datou_step_exec : 41.88737607002258 time spend to save output : 0.04264020919799805 total time spend for step 6 : 41.93001627922058 step7:ventilate_hashtags_in_portfolio Wed Feb 12 12:32:46 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20476171 get user id for portfolio 20476171 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20476171 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('flou','Teint_Dans_La_Masse','cartonnette','plastique','metal','kraft','papier','environnement','autre_refus','Carton_brun','Carton_gris','mal_croppe')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20476171 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('flou','Teint_Dans_La_Masse','cartonnette','plastique','metal','kraft','papier','environnement','autre_refus','Carton_brun','Carton_gris','mal_croppe')) AND mptpi.`min_score`=0.5 To do Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") Catched exception ! Connect or reconnect ! (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')\n and cspi.crop_hashtag_id = chi.id' at line 3") To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20476171 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('flou','Teint_Dans_La_Masse','cartonnette','plastique','metal','kraft','papier','environnement','autre_refus','Carton_brun','Carton_gris','mal_croppe')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20476400,20476401,20476402,20476403,20476404,20476405,20476406,20476407,20476408,20476409,20476410,20476411?tags=flou,Teint_Dans_La_Masse,cartonnette,plastique,metal,kraft,papier,environnement,autre_refus,Carton_brun,Carton_gris,mal_croppe Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1335238113, 1335237999, 1335237967, 1335237931, 1335237877, 1335237738, 1335237734, 1335237716, 1335237690, 1335237669] Looping around the photos to save general results len do output : 1 /20476171. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335238113', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237999', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237967', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237931', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237877', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237738', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237734', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237716', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237690', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237669', None, None, None, None, None, '2579320') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.014104127883911133 save_final save missing photos in datou_result : time spend for datou_step_exec : 3.4774441719055176 time spend to save output : 0.014605998992919922 total time spend for step 7 : 3.4920501708984375 step8:final Wed Feb 12 12:32:50 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1335238113: ('0.09647819935167498',), 1335237999: ('0.09647819935167498',), 1335237967: ('0.09647819935167498',), 1335237931: ('0.09647819935167498',), 1335237877: ('0.09647819935167498',), 1335237738: ('0.09647819935167498',), 1335237734: ('0.09647819935167498',), 1335237716: ('0.09647819935167498',), 1335237690: ('0.09647819935167498',), 1335237669: ('0.09647819935167498',)} new output for save of step final : {1335238113: ('0.09647819935167498',), 1335237999: ('0.09647819935167498',), 1335237967: ('0.09647819935167498',), 1335237931: ('0.09647819935167498',), 1335237877: ('0.09647819935167498',), 1335237738: ('0.09647819935167498',), 1335237734: ('0.09647819935167498',), 1335237716: ('0.09647819935167498',), 1335237690: ('0.09647819935167498',), 1335237669: ('0.09647819935167498',)} [1335238113, 1335237999, 1335237967, 1335237931, 1335237877, 1335237738, 1335237734, 1335237716, 1335237690, 1335237669] Looping around the photos to save general results len do output : 10 /1335238113.Didn't retrieve data . /1335237999.Didn't retrieve data . /1335237967.Didn't retrieve data . /1335237931.Didn't retrieve data . /1335237877.Didn't retrieve data . /1335237738.Didn't retrieve data . /1335237734.Didn't retrieve data . /1335237716.Didn't retrieve data . /1335237690.Didn't retrieve data . /1335237669.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335238113', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237999', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237967', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237931', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237877', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237738', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237734', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237716', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237690', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237669', None, None, None, None, None, '2579320') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013481616973876953 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.14506912231445312 time spend to save output : 0.014027595520019531 total time spend for step 8 : 0.15909671783447266 step9:velours_tree Wed Feb 12 12:32:50 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.07907724380493164 time spend to save output : 3.981590270996094e-05 total time spend for step 9 : 0.0791170597076416 step10:send_mail_cod Wed Feb 12 12:32:50 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20476171_12-02-2025_12_32_50.pdf 20476400 imagette204764001739359970 20476402 change filename to text .change filename to text .imagette204764021739359970 20476403 change filename to text .change filename to text .imagette204764031739359970 20476404 imagette204764041739359971 20476405 imagette204764051739359971 20476408 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette204764081739359971 20476411 imagette204764111739359971 20476401 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette204764011739359971 20476409 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette204764091739359972 20476410 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette204764101739359972 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20476171 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20476400,20476401,20476402,20476403,20476404,20476405,20476406,20476407,20476408,20476409,20476410,20476411?tags=flou,Teint_Dans_La_Masse,cartonnette,plastique,metal,kraft,papier,environnement,autre_refus,Carton_brun,Carton_gris,mal_croppe your option no_mail is active, we will not send the real mail to your client args[1335238113] : ('0.09647819935167498',) no score found for photo 1335238113 apple ('0.09647819935167498',) We are sending mail with results at report@fotonower.com args[1335237999] : ('0.09647819935167498',) no score found for photo 1335237999 apple ('0.09647819935167498',) We are sending mail with results at report@fotonower.com args[1335237967] : ('0.09647819935167498',) no score found for photo 1335237967 apple ('0.09647819935167498',) We are sending mail with results at report@fotonower.com args[1335237931] : ('0.09647819935167498',) no score found for photo 1335237931 apple ('0.09647819935167498',) We are sending mail with results at report@fotonower.com args[1335237877] : ('0.09647819935167498',) no score found for photo 1335237877 apple ('0.09647819935167498',) We are sending mail with results at report@fotonower.com args[1335237738] : ('0.09647819935167498',) no score found for photo 1335237738 apple ('0.09647819935167498',) We are sending mail with results at report@fotonower.com args[1335237734] : ('0.09647819935167498',) no score found for photo 1335237734 apple ('0.09647819935167498',) We are sending mail with results at report@fotonower.com args[1335237716] : ('0.09647819935167498',) no score found for photo 1335237716 apple ('0.09647819935167498',) We are sending mail with results at report@fotonower.com args[1335237690] : ('0.09647819935167498',) no score found for photo 1335237690 apple ('0.09647819935167498',) We are sending mail with results at report@fotonower.com args[1335237669] : ('0.09647819935167498',) no score found for photo 1335237669 apple ('0.09647819935167498',) We are sending mail with results at report@fotonower.com refus_total : 0.09647819935167498 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20476171 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1335237474,1335237716,1335237734,1335237738,1335237931,1335237967,1335237612,1335237669,1335237465,1335237522,1335237690,1335237877,1335237999,1335238113) Found this number of photos: 14 begin to download photo : 1335237474 begin to download photo : 1335237738 begin to download photo : 1335237612 begin to download photo : 1335237522 begin to download photo : 1335237999 download finish for photo 1335237474 begin to download photo : 1335237716 download finish for photo 1335237612 begin to download photo : 1335237669 download finish for photo 1335237522 begin to download photo : 1335237690 download finish for photo 1335237999 begin to download photo : 1335238113 download finish for photo 1335237738 begin to download photo : 1335237931 download finish for photo 1335238113 download finish for photo 1335237716 begin to download photo : 1335237734 download finish for photo 1335237669 begin to download photo : 1335237465 download finish for photo 1335237690 begin to download photo : 1335237877 download finish for photo 1335237931 begin to download photo : 1335237967 download finish for photo 1335237734 download finish for photo 1335237465 download finish for photo 1335237877 download finish for photo 1335237967 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20476171_12-02-2025_12_32_50.pdf results_Auto_P20476171_12-02-2025_12_32_50.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20476171_12-02-2025_12_32_50.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20476171','results_Auto_P20476171_12-02-2025_12_32_50.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20476171_12-02-2025_12_32_50.pdf','pdf','','0.83','0.09647819935167498') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1335238113, 1335237999, 1335237967, 1335237931, 1335237877, 1335237738, 1335237734, 1335237716, 1335237690, 1335237669] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335238113', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237999', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237967', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237931', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237877', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237738', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237734', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237716', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237690', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237669', None, None, None, None, None, '2579320') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.01304173469543457 save_final save missing photos in datou_result : time spend for datou_step_exec : 6.468756914138794 time spend to save output : 0.013286352157592773 total time spend for step 10 : 6.482043266296387 step11:split_time_score Wed Feb 12 12:32:57 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('17', 14),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 06022025 20476171 Nombre de photos uploadées : 14 / 23040 (0%) 06022025 20476171 Nombre de photos taguées (types de déchets): 0 / 14 (0%) 06022025 20476171 Nombre de photos taguées (volume) : 0 / 14 (0%) elapsed_time : load_data_split_time_score 1.6689300537109375e-06 elapsed_time : order_list_meta_photo_and_scores 5.0067901611328125e-06 ?????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0007913112640380859 elapsed_time : insert_dashboard_record_day_entry 0.043630123138427734 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps Qualite : 0.09647819935167498 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20476171_12-02-2025_12_32_50.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20476171 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20476171 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20476172 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20303853 order by id desc limit 1 Qualite : 0.03632571502580891 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20303854_06-02-2025_20_45_33.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20303854 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20303854 AND mptpi.`type`=4207 To do Qualite : 0.08488419576320486 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20303855_06-02-2025_20_19_55.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20303855 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11500 mask_detect is not consistent : 3 used against 2 in the step definition ! WARNING : number of outputs for step 11508 brightness is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11509 blur_detection is not consistent : 2 used against 1 in the step definition ! WARNING : number of inputs for step 11504 crop_condition is not consistent : 3 used against 2 in the step definition ! Step 11504 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11507 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11507 merge_mask_thcl_custom is not consistent : 3 used against 2 in the step definition ! WARNING : number of inputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11501 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11576 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11576 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of outputs for step 11503 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11502 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11502 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11511 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 11503 doesn't seem to be define in the database( WARNING : type of input 3 of step 11502 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 2 of step 11500 doesn't seem to be define in the database( WARNING : type of input 2 of step 11504 doesn't seem to be define in the database( WARNING : output 1 of step 11500 have datatype=2 whereas input 1 of step 11507 have datatype=7 WARNING : type of output 2 of step 11507 doesn't seem to be define in the database( WARNING : type of input 1 of step 11501 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11503 have datatype=10 whereas input 3 of step 11510 have datatype=6 WARNING : type of input 2 of step 11576 doesn't seem to be define in the database( WARNING : output 1 of step 11501 have datatype=7 whereas input 2 of step 11576 have datatype=None WARNING : type of output 3 of step 11576 doesn't seem to be define in the database( WARNING : type of input 1 of step 11503 doesn't seem to be define in the database( WARNING : output 0 of step 11503 have datatype=10 whereas input 0 of step 11582 have datatype=18 WARNING : type of input 5 of step 11510 doesn't seem to be define in the database( WARNING : output 0 of step 11582 have datatype=11 whereas input 5 of step 11510 have datatype=None WARNING : type of output 1 of step 11508 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : type of output 1 of step 11509 doesn't seem to be define in the database( WARNING : type of input 3 of step 11504 doesn't seem to be define in the database( WARNING : output 0 of step 11507 have datatype=1 whereas input 0 of step 11501 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20303855 AND mptpi.`type`=4207 To do NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'06022025': {'nb_upload': 14, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1335238113, 1335237999, 1335237967, 1335237931, 1335237877, 1335237738, 1335237734, 1335237716, 1335237690, 1335237669] Looping around the photos to save general results len do output : 1 /20476171Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335238113', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237999', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237967', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237931', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237877', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237738', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237734', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237716', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237690', None, None, None, None, None, '2579320') ('3459', None, None, None, None, None, None, None, '2579320') ('3459', '20476171', '1335237669', None, None, None, None, None, '2579320') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.01336526870727539 save_final save missing photos in datou_result : time spend for datou_step_exec : 1.9421422481536865 time spend to save output : 0.013575077056884766 total time spend for step 11 : 1.9557173252105713 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 172.24user 243.54system 8:34.07elapsed 80%CPU (0avgtext+0avgdata 4238400maxresident)k 323464inputs+147952outputs (7357major+17115764minor)pagefaults 0swaps