python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 929578 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 22>10 please execute split_portfolio.py -i 20314827 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2560885'] with mtr_portfolio_ids : ['20314827'] and first list_photo_ids : [] new path : /proc/929578/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 1.9805009365081787 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Fri Feb 7 01:34:30 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 10772 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-07 01:34:36.933201: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-07 01:34:36.967089: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-07 01:34:36.969186: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f98b8000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-07 01:34:36.969235: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-07 01:34:36.974697: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-07 01:34:37.113126: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x42d86630 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-07 01:34:37.113170: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-07 01:34:37.114951: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-07 01:34:37.115338: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-07 01:34:37.118123: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-07 01:34:37.138480: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-07 01:34:37.138843: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-07 01:34:37.175070: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-07 01:34:37.180743: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-07 01:34:37.240555: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-07 01:34:37.242490: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-07 01:34:37.242951: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-07 01:34:37.244262: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-07 01:34:37.244283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-07 01:34:37.244295: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-07 01:34:37.247131: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9777 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-07 01:34:37.560831: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-07 01:34:37.560967: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-07 01:34:37.560991: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-07 01:34:37.561011: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-07 01:34:37.561031: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-07 01:34:37.561050: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-07 01:34:37.561069: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-07 01:34:37.561089: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-07 01:34:37.562763: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-07 01:34:37.564163: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-07 01:34:37.564202: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-07 01:34:37.564221: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-07 01:34:37.564238: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-07 01:34:37.564255: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-07 01:34:37.564272: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-07 01:34:37.564289: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-07 01:34:37.564305: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-07 01:34:37.566005: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-07 01:34:37.566044: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-07 01:34:37.566055: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-07 01:34:37.566066: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-07 01:34:37.567920: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9777 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-07 01:34:49.086599: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-07 01:34:49.292953: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 22 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 29 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 25 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 14 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 34 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 31 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 33 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 14 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 15 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 21 Detection mask done ! Trying to reset tf kernel 929978 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5262 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 10551 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.0003001689910888672 nb_pixel_total : 8268 time to create 1 rle with old method : 0.009580850601196289 length of segment : 110 time for calcul the mask position with numpy : 0.00032901763916015625 nb_pixel_total : 7320 time to create 1 rle with old method : 0.008883476257324219 length of segment : 181 time for calcul the mask position with numpy : 0.005532741546630859 nb_pixel_total : 187448 time to create 1 rle with new method : 0.019805431365966797 length of segment : 978 time for calcul the mask position with numpy : 0.0011360645294189453 nb_pixel_total : 68545 time to create 1 rle with old method : 0.08034658432006836 length of segment : 261 time for calcul the mask position with numpy : 0.0006470680236816406 nb_pixel_total : 35346 time to create 1 rle with old method : 0.0422511100769043 length of segment : 293 time for calcul the mask position with numpy : 0.00014710426330566406 nb_pixel_total : 5880 time to create 1 rle with old method : 0.007353067398071289 length of segment : 67 time for calcul the mask position with numpy : 0.014054059982299805 nb_pixel_total : 720042 time to create 1 rle with new method : 0.03476142883300781 length of segment : 1339 time for calcul the mask position with numpy : 0.0002124309539794922 nb_pixel_total : 11519 time to create 1 rle with old method : 0.01379251480102539 length of segment : 115 time for calcul the mask position with numpy : 0.03195452690124512 nb_pixel_total : 1417967 time to create 1 rle with new method : 0.11371898651123047 length of segment : 2159 time for calcul the mask position with numpy : 0.0022628307342529297 nb_pixel_total : 86810 time to create 1 rle with old method : 0.09675192832946777 length of segment : 398 time for calcul the mask position with numpy : 0.01877117156982422 nb_pixel_total : 520788 time to create 1 rle with new method : 0.06347990036010742 length of segment : 1252 time for calcul the mask position with numpy : 0.0005459785461425781 nb_pixel_total : 27481 time to create 1 rle with old method : 0.03166794776916504 length of segment : 139 time for calcul the mask position with numpy : 0.0035772323608398438 nb_pixel_total : 150836 time to create 1 rle with new method : 0.011755228042602539 length of segment : 429 time for calcul the mask position with numpy : 8.893013000488281e-05 nb_pixel_total : 1256 time to create 1 rle with old method : 0.0016274452209472656 length of segment : 64 time for calcul the mask position with numpy : 0.00023317337036132812 nb_pixel_total : 9758 time to create 1 rle with old method : 0.011304855346679688 length of segment : 133 time for calcul the mask position with numpy : 0.0004191398620605469 nb_pixel_total : 20373 time to create 1 rle with old method : 0.022024154663085938 length of segment : 160 time for calcul the mask position with numpy : 0.0026793479919433594 nb_pixel_total : 156986 time to create 1 rle with new method : 0.008507251739501953 length of segment : 856 time for calcul the mask position with numpy : 0.00342559814453125 nb_pixel_total : 216366 time to create 1 rle with new method : 0.01009058952331543 length of segment : 626 time for calcul the mask position with numpy : 0.0002944469451904297 nb_pixel_total : 6048 time to create 1 rle with old method : 0.007227659225463867 length of segment : 118 time for calcul the mask position with numpy : 0.0010082721710205078 nb_pixel_total : 45470 time to create 1 rle with old method : 0.0515131950378418 length of segment : 302 time for calcul the mask position with numpy : 0.0032165050506591797 nb_pixel_total : 134712 time to create 1 rle with old method : 0.1462383270263672 length of segment : 542 time for calcul the mask position with numpy : 0.0031244754791259766 nb_pixel_total : 167853 time to create 1 rle with new method : 0.011101245880126953 length of segment : 434 time for calcul the mask position with numpy : 0.00018978118896484375 nb_pixel_total : 6316 time to create 1 rle with old method : 0.007534027099609375 length of segment : 85 time for calcul the mask position with numpy : 0.0003571510314941406 nb_pixel_total : 18369 time to create 1 rle with old method : 0.022051572799682617 length of segment : 154 time for calcul the mask position with numpy : 8.130073547363281e-05 nb_pixel_total : 1428 time to create 1 rle with old method : 0.0018086433410644531 length of segment : 50 time for calcul the mask position with numpy : 0.0006556510925292969 nb_pixel_total : 17089 time to create 1 rle with old method : 0.019939899444580078 length of segment : 328 time for calcul the mask position with numpy : 0.0019118785858154297 nb_pixel_total : 58051 time to create 1 rle with old method : 0.06442689895629883 length of segment : 493 time for calcul the mask position with numpy : 0.0001399517059326172 nb_pixel_total : 2382 time to create 1 rle with old method : 0.0029120445251464844 length of segment : 105 time for calcul the mask position with numpy : 0.004786491394042969 nb_pixel_total : 258804 time to create 1 rle with new method : 0.013876914978027344 length of segment : 638 time for calcul the mask position with numpy : 0.0003657341003417969 nb_pixel_total : 11346 time to create 1 rle with old method : 0.012693405151367188 length of segment : 260 time for calcul the mask position with numpy : 0.003284931182861328 nb_pixel_total : 121622 time to create 1 rle with old method : 0.13334059715270996 length of segment : 512 time for calcul the mask position with numpy : 0.00441288948059082 nb_pixel_total : 228042 time to create 1 rle with new method : 0.012954235076904297 length of segment : 608 time for calcul the mask position with numpy : 0.002735137939453125 nb_pixel_total : 161305 time to create 1 rle with new method : 0.008819341659545898 length of segment : 454 time for calcul the mask position with numpy : 0.00021147727966308594 nb_pixel_total : 4199 time to create 1 rle with old method : 0.005522489547729492 length of segment : 95 time for calcul the mask position with numpy : 0.00014853477478027344 nb_pixel_total : 4828 time to create 1 rle with old method : 0.005975484848022461 length of segment : 72 time for calcul the mask position with numpy : 0.0009684562683105469 nb_pixel_total : 3692 time to create 1 rle with old method : 0.004744529724121094 length of segment : 115 time for calcul the mask position with numpy : 0.0006084442138671875 nb_pixel_total : 24750 time to create 1 rle with old method : 0.028617382049560547 length of segment : 258 time for calcul the mask position with numpy : 0.00017404556274414062 nb_pixel_total : 2114 time to create 1 rle with old method : 0.002802133560180664 length of segment : 120 time for calcul the mask position with numpy : 0.0009188652038574219 nb_pixel_total : 44436 time to create 1 rle with old method : 0.05137801170349121 length of segment : 265 time for calcul the mask position with numpy : 0.00342559814453125 nb_pixel_total : 181398 time to create 1 rle with new method : 0.011148452758789062 length of segment : 794 time for calcul the mask position with numpy : 0.0021278858184814453 nb_pixel_total : 110270 time to create 1 rle with old method : 0.12252593040466309 length of segment : 387 time for calcul the mask position with numpy : 0.0013146400451660156 nb_pixel_total : 66528 time to create 1 rle with old method : 0.0730438232421875 length of segment : 329 time for calcul the mask position with numpy : 0.00017642974853515625 nb_pixel_total : 5807 time to create 1 rle with old method : 0.00664067268371582 length of segment : 118 time for calcul the mask position with numpy : 0.002642393112182617 nb_pixel_total : 128799 time to create 1 rle with old method : 0.15285944938659668 length of segment : 461 time for calcul the mask position with numpy : 0.0001323223114013672 nb_pixel_total : 3349 time to create 1 rle with old method : 0.004169464111328125 length of segment : 55 time for calcul the mask position with numpy : 0.002454519271850586 nb_pixel_total : 112372 time to create 1 rle with old method : 0.14368677139282227 length of segment : 374 time for calcul the mask position with numpy : 0.0002503395080566406 nb_pixel_total : 5697 time to create 1 rle with old method : 0.007827043533325195 length of segment : 86 time for calcul the mask position with numpy : 0.0036821365356445312 nb_pixel_total : 192091 time to create 1 rle with new method : 0.010904312133789062 length of segment : 552 time for calcul the mask position with numpy : 0.0012555122375488281 nb_pixel_total : 42934 time to create 1 rle with old method : 0.0554044246673584 length of segment : 274 time for calcul the mask position with numpy : 0.0033991336822509766 nb_pixel_total : 174938 time to create 1 rle with new method : 0.009886980056762695 length of segment : 544 time for calcul the mask position with numpy : 0.001035451889038086 nb_pixel_total : 43978 time to create 1 rle with old method : 0.05645346641540527 length of segment : 261 time for calcul the mask position with numpy : 0.0007445812225341797 nb_pixel_total : 29665 time to create 1 rle with old method : 0.037648916244506836 length of segment : 187 time for calcul the mask position with numpy : 0.0019860267639160156 nb_pixel_total : 86914 time to create 1 rle with old method : 0.10419034957885742 length of segment : 795 time for calcul the mask position with numpy : 0.00020360946655273438 nb_pixel_total : 4462 time to create 1 rle with old method : 0.0054242610931396484 length of segment : 78 time for calcul the mask position with numpy : 0.0029811859130859375 nb_pixel_total : 160676 time to create 1 rle with new method : 0.00971364974975586 length of segment : 639 time for calcul the mask position with numpy : 0.007441520690917969 nb_pixel_total : 377758 time to create 1 rle with new method : 0.01907944679260254 length of segment : 567 time for calcul the mask position with numpy : 0.0003733634948730469 nb_pixel_total : 11466 time to create 1 rle with old method : 0.015462160110473633 length of segment : 118 time for calcul the mask position with numpy : 0.0017447471618652344 nb_pixel_total : 48048 time to create 1 rle with old method : 0.08231067657470703 length of segment : 317 time for calcul the mask position with numpy : 0.011856555938720703 nb_pixel_total : 313386 time to create 1 rle with new method : 0.016448259353637695 length of segment : 417 time for calcul the mask position with numpy : 0.00028204917907714844 nb_pixel_total : 1661 time to create 1 rle with old method : 0.0028204917907714844 length of segment : 29 time for calcul the mask position with numpy : 0.0006513595581054688 nb_pixel_total : 7305 time to create 1 rle with old method : 0.011232852935791016 length of segment : 120 time for calcul the mask position with numpy : 0.002841472625732422 nb_pixel_total : 58675 time to create 1 rle with old method : 0.08012056350708008 length of segment : 387 time for calcul the mask position with numpy : 0.0006074905395507812 nb_pixel_total : 4759 time to create 1 rle with old method : 0.0075206756591796875 length of segment : 59 time for calcul the mask position with numpy : 0.04267549514770508 nb_pixel_total : 1329058 time to create 1 rle with new method : 0.8003411293029785 length of segment : 2024 time for calcul the mask position with numpy : 0.0015931129455566406 nb_pixel_total : 73535 time to create 1 rle with old method : 0.08483529090881348 length of segment : 419 time for calcul the mask position with numpy : 0.00022220611572265625 nb_pixel_total : 2589 time to create 1 rle with old method : 0.003769397735595703 length of segment : 90 time for calcul the mask position with numpy : 0.00016951560974121094 nb_pixel_total : 4746 time to create 1 rle with old method : 0.006638288497924805 length of segment : 64 time for calcul the mask position with numpy : 0.01922321319580078 nb_pixel_total : 751973 time to create 1 rle with new method : 0.07290244102478027 length of segment : 1078 time for calcul the mask position with numpy : 0.00017881393432617188 nb_pixel_total : 4003 time to create 1 rle with old method : 0.004858255386352539 length of segment : 99 time for calcul the mask position with numpy : 0.00042724609375 nb_pixel_total : 25794 time to create 1 rle with old method : 0.03165698051452637 length of segment : 214 time for calcul the mask position with numpy : 9.012222290039062e-05 nb_pixel_total : 2788 time to create 1 rle with old method : 0.0035452842712402344 length of segment : 87 time for calcul the mask position with numpy : 0.00034356117248535156 nb_pixel_total : 7889 time to create 1 rle with old method : 0.009823083877563477 length of segment : 208 time for calcul the mask position with numpy : 0.00012445449829101562 nb_pixel_total : 3112 time to create 1 rle with old method : 0.004156589508056641 length of segment : 53 time for calcul the mask position with numpy : 0.0005915164947509766 nb_pixel_total : 8782 time to create 1 rle with old method : 0.01066279411315918 length of segment : 134 time for calcul the mask position with numpy : 0.0051517486572265625 nb_pixel_total : 158116 time to create 1 rle with new method : 0.008520126342773438 length of segment : 561 time for calcul the mask position with numpy : 0.0016722679138183594 nb_pixel_total : 58271 time to create 1 rle with old method : 0.06731939315795898 length of segment : 289 time for calcul the mask position with numpy : 0.0027992725372314453 nb_pixel_total : 73355 time to create 1 rle with old method : 0.08752799034118652 length of segment : 385 time for calcul the mask position with numpy : 0.002791881561279297 nb_pixel_total : 79751 time to create 1 rle with old method : 0.08683371543884277 length of segment : 608 time for calcul the mask position with numpy : 0.0002434253692626953 nb_pixel_total : 4410 time to create 1 rle with old method : 0.005320072174072266 length of segment : 44 time for calcul the mask position with numpy : 0.0014650821685791016 nb_pixel_total : 48870 time to create 1 rle with old method : 0.05847501754760742 length of segment : 319 time for calcul the mask position with numpy : 0.0005166530609130859 nb_pixel_total : 9046 time to create 1 rle with old method : 0.011990785598754883 length of segment : 93 time for calcul the mask position with numpy : 0.0023419857025146484 nb_pixel_total : 65494 time to create 1 rle with old method : 0.07740998268127441 length of segment : 389 time for calcul the mask position with numpy : 0.0005230903625488281 nb_pixel_total : 11470 time to create 1 rle with old method : 0.013754129409790039 length of segment : 141 time for calcul the mask position with numpy : 0.006562232971191406 nb_pixel_total : 203624 time to create 1 rle with new method : 0.01041722297668457 length of segment : 610 time for calcul the mask position with numpy : 0.0008885860443115234 nb_pixel_total : 20069 time to create 1 rle with old method : 0.02317523956298828 length of segment : 284 time for calcul the mask position with numpy : 0.0001277923583984375 nb_pixel_total : 1536 time to create 1 rle with old method : 0.0019989013671875 length of segment : 88 time for calcul the mask position with numpy : 0.00051116943359375 nb_pixel_total : 9070 time to create 1 rle with old method : 0.010483026504516602 length of segment : 151 time for calcul the mask position with numpy : 0.000713348388671875 nb_pixel_total : 5746 time to create 1 rle with old method : 0.007114887237548828 length of segment : 161 time for calcul the mask position with numpy : 0.0009245872497558594 nb_pixel_total : 19967 time to create 1 rle with old method : 0.02396702766418457 length of segment : 184 time for calcul the mask position with numpy : 0.0002703666687011719 nb_pixel_total : 6717 time to create 1 rle with old method : 0.008443117141723633 length of segment : 134 time for calcul the mask position with numpy : 0.002202749252319336 nb_pixel_total : 58795 time to create 1 rle with old method : 0.07014727592468262 length of segment : 290 time for calcul the mask position with numpy : 0.002045154571533203 nb_pixel_total : 54479 time to create 1 rle with old method : 0.06443071365356445 length of segment : 416 time for calcul the mask position with numpy : 0.0005941390991210938 nb_pixel_total : 10049 time to create 1 rle with old method : 0.012195110321044922 length of segment : 237 time for calcul the mask position with numpy : 0.0004451274871826172 nb_pixel_total : 5345 time to create 1 rle with old method : 0.006827592849731445 length of segment : 149 time for calcul the mask position with numpy : 0.003392457962036133 nb_pixel_total : 74109 time to create 1 rle with old method : 0.0853109359741211 length of segment : 346 time for calcul the mask position with numpy : 0.0005214214324951172 nb_pixel_total : 11852 time to create 1 rle with old method : 0.014370203018188477 length of segment : 126 time for calcul the mask position with numpy : 0.0011110305786132812 nb_pixel_total : 28914 time to create 1 rle with old method : 0.03487229347229004 length of segment : 147 time for calcul the mask position with numpy : 0.0004298686981201172 nb_pixel_total : 13429 time to create 1 rle with old method : 0.016396760940551758 length of segment : 126 time for calcul the mask position with numpy : 0.00017952919006347656 nb_pixel_total : 1480 time to create 1 rle with old method : 0.0020325183868408203 length of segment : 51 time for calcul the mask position with numpy : 0.0017881393432617188 nb_pixel_total : 65084 time to create 1 rle with old method : 0.07252001762390137 length of segment : 401 time for calcul the mask position with numpy : 0.0011258125305175781 nb_pixel_total : 19231 time to create 1 rle with old method : 0.022130966186523438 length of segment : 332 time for calcul the mask position with numpy : 0.0013828277587890625 nb_pixel_total : 20841 time to create 1 rle with old method : 0.024970531463623047 length of segment : 231 time for calcul the mask position with numpy : 0.001123189926147461 nb_pixel_total : 26638 time to create 1 rle with old method : 0.03160524368286133 length of segment : 245 time for calcul the mask position with numpy : 0.0005886554718017578 nb_pixel_total : 10847 time to create 1 rle with old method : 0.013455867767333984 length of segment : 187 time for calcul the mask position with numpy : 0.0005908012390136719 nb_pixel_total : 11537 time to create 1 rle with old method : 0.01437687873840332 length of segment : 148 time for calcul the mask position with numpy : 0.004908323287963867 nb_pixel_total : 154785 time to create 1 rle with new method : 0.007086753845214844 length of segment : 556 time for calcul the mask position with numpy : 0.006956815719604492 nb_pixel_total : 202390 time to create 1 rle with new method : 0.011366128921508789 length of segment : 616 time for calcul the mask position with numpy : 0.00026679039001464844 nb_pixel_total : 3337 time to create 1 rle with old method : 0.004372596740722656 length of segment : 44 time for calcul the mask position with numpy : 0.0018563270568847656 nb_pixel_total : 53930 time to create 1 rle with old method : 0.061563968658447266 length of segment : 242 time for calcul the mask position with numpy : 0.003201723098754883 nb_pixel_total : 89326 time to create 1 rle with old method : 0.10250139236450195 length of segment : 541 time for calcul the mask position with numpy : 0.0015413761138916016 nb_pixel_total : 45688 time to create 1 rle with old method : 0.052599430084228516 length of segment : 284 time for calcul the mask position with numpy : 0.002264738082885742 nb_pixel_total : 70841 time to create 1 rle with old method : 0.08159470558166504 length of segment : 497 time for calcul the mask position with numpy : 0.00012993812561035156 nb_pixel_total : 1219 time to create 1 rle with old method : 0.001653432846069336 length of segment : 46 time for calcul the mask position with numpy : 0.0009553432464599609 nb_pixel_total : 25825 time to create 1 rle with old method : 0.0309903621673584 length of segment : 381 time for calcul the mask position with numpy : 0.0001289844512939453 nb_pixel_total : 1558 time to create 1 rle with old method : 0.0021266937255859375 length of segment : 89 time for calcul the mask position with numpy : 0.0004935264587402344 nb_pixel_total : 11665 time to create 1 rle with old method : 0.014178752899169922 length of segment : 125 time for calcul the mask position with numpy : 0.0019176006317138672 nb_pixel_total : 50746 time to create 1 rle with old method : 0.060585737228393555 length of segment : 373 time for calcul the mask position with numpy : 0.0004630088806152344 nb_pixel_total : 5440 time to create 1 rle with old method : 0.007103681564331055 length of segment : 127 time for calcul the mask position with numpy : 0.0024335384368896484 nb_pixel_total : 79753 time to create 1 rle with old method : 0.09238886833190918 length of segment : 389 time for calcul the mask position with numpy : 0.0018203258514404297 nb_pixel_total : 83275 time to create 1 rle with old method : 0.11943769454956055 length of segment : 343 time for calcul the mask position with numpy : 0.0002009868621826172 nb_pixel_total : 2101 time to create 1 rle with old method : 0.00262451171875 length of segment : 89 time for calcul the mask position with numpy : 0.0001914501190185547 nb_pixel_total : 7403 time to create 1 rle with old method : 0.009455442428588867 length of segment : 140 time for calcul the mask position with numpy : 0.0012023448944091797 nb_pixel_total : 28232 time to create 1 rle with old method : 0.03362083435058594 length of segment : 256 time for calcul the mask position with numpy : 0.00036406517028808594 nb_pixel_total : 4802 time to create 1 rle with old method : 0.005662441253662109 length of segment : 113 time for calcul the mask position with numpy : 0.001195669174194336 nb_pixel_total : 23256 time to create 1 rle with old method : 0.02696537971496582 length of segment : 207 time for calcul the mask position with numpy : 0.0006175041198730469 nb_pixel_total : 8961 time to create 1 rle with old method : 0.010874271392822266 length of segment : 134 time for calcul the mask position with numpy : 0.0008311271667480469 nb_pixel_total : 21628 time to create 1 rle with old method : 0.0259702205657959 length of segment : 208 time for calcul the mask position with numpy : 0.00015735626220703125 nb_pixel_total : 1510 time to create 1 rle with old method : 0.0019676685333251953 length of segment : 47 time for calcul the mask position with numpy : 0.0015006065368652344 nb_pixel_total : 73019 time to create 1 rle with old method : 0.08064031600952148 length of segment : 432 time for calcul the mask position with numpy : 0.0003516674041748047 nb_pixel_total : 8998 time to create 1 rle with old method : 0.010689020156860352 length of segment : 143 time for calcul the mask position with numpy : 0.0015163421630859375 nb_pixel_total : 22835 time to create 1 rle with old method : 0.026915788650512695 length of segment : 207 time for calcul the mask position with numpy : 0.0007386207580566406 nb_pixel_total : 15999 time to create 1 rle with old method : 0.019517183303833008 length of segment : 260 time for calcul the mask position with numpy : 0.0017020702362060547 nb_pixel_total : 48682 time to create 1 rle with old method : 0.058432817459106445 length of segment : 324 time for calcul the mask position with numpy : 0.0002543926239013672 nb_pixel_total : 2564 time to create 1 rle with old method : 0.0032072067260742188 length of segment : 83 time for calcul the mask position with numpy : 0.00026988983154296875 nb_pixel_total : 5238 time to create 1 rle with old method : 0.00655674934387207 length of segment : 75 time for calcul the mask position with numpy : 0.0003101825714111328 nb_pixel_total : 5832 time to create 1 rle with old method : 0.007454872131347656 length of segment : 72 time for calcul the mask position with numpy : 0.0001647472381591797 nb_pixel_total : 3538 time to create 1 rle with old method : 0.004445552825927734 length of segment : 98 time for calcul the mask position with numpy : 0.0020935535430908203 nb_pixel_total : 30145 time to create 1 rle with old method : 0.03489494323730469 length of segment : 331 time for calcul the mask position with numpy : 0.0006330013275146484 nb_pixel_total : 10898 time to create 1 rle with old method : 0.013567209243774414 length of segment : 144 time for calcul the mask position with numpy : 0.002022266387939453 nb_pixel_total : 41479 time to create 1 rle with old method : 0.04754018783569336 length of segment : 221 time for calcul the mask position with numpy : 0.003269195556640625 nb_pixel_total : 90933 time to create 1 rle with old method : 0.10564684867858887 length of segment : 356 time for calcul the mask position with numpy : 0.0014848709106445312 nb_pixel_total : 58823 time to create 1 rle with old method : 0.06773996353149414 length of segment : 309 time for calcul the mask position with numpy : 0.002565622329711914 nb_pixel_total : 46865 time to create 1 rle with old method : 0.053801774978637695 length of segment : 299 time for calcul the mask position with numpy : 0.0004868507385253906 nb_pixel_total : 6756 time to create 1 rle with old method : 0.008708953857421875 length of segment : 58 time for calcul the mask position with numpy : 0.0038993358612060547 nb_pixel_total : 106982 time to create 1 rle with old method : 0.12630534172058105 length of segment : 390 time for calcul the mask position with numpy : 0.000186920166015625 nb_pixel_total : 1340 time to create 1 rle with old method : 0.0017895698547363281 length of segment : 43 time for calcul the mask position with numpy : 0.0030167102813720703 nb_pixel_total : 78402 time to create 1 rle with old method : 0.08724784851074219 length of segment : 456 time for calcul the mask position with numpy : 0.005686759948730469 nb_pixel_total : 162985 time to create 1 rle with new method : 0.009469747543334961 length of segment : 673 time for calcul the mask position with numpy : 0.0007798671722412109 nb_pixel_total : 19534 time to create 1 rle with old method : 0.023017168045043945 length of segment : 229 time for calcul the mask position with numpy : 0.0002498626708984375 nb_pixel_total : 2472 time to create 1 rle with old method : 0.0030007362365722656 length of segment : 95 time for calcul the mask position with numpy : 0.0002925395965576172 nb_pixel_total : 3038 time to create 1 rle with old method : 0.003927946090698242 length of segment : 87 time for calcul the mask position with numpy : 0.0001735687255859375 nb_pixel_total : 1834 time to create 1 rle with old method : 0.002424001693725586 length of segment : 50 time for calcul the mask position with numpy : 0.0007765293121337891 nb_pixel_total : 13538 time to create 1 rle with old method : 0.016515493392944336 length of segment : 89 time for calcul the mask position with numpy : 0.0002830028533935547 nb_pixel_total : 5437 time to create 1 rle with old method : 0.006657600402832031 length of segment : 81 time for calcul the mask position with numpy : 0.0005424022674560547 nb_pixel_total : 20454 time to create 1 rle with old method : 0.023194074630737305 length of segment : 252 time for calcul the mask position with numpy : 0.0005893707275390625 nb_pixel_total : 11435 time to create 1 rle with old method : 0.014007091522216797 length of segment : 135 time for calcul the mask position with numpy : 0.0016553401947021484 nb_pixel_total : 97537 time to create 1 rle with old method : 0.11080622673034668 length of segment : 349 time for calcul the mask position with numpy : 0.0005450248718261719 nb_pixel_total : 5223 time to create 1 rle with old method : 0.006451129913330078 length of segment : 158 time for calcul the mask position with numpy : 0.0013823509216308594 nb_pixel_total : 41142 time to create 1 rle with old method : 0.04829883575439453 length of segment : 270 time for calcul the mask position with numpy : 0.0003943443298339844 nb_pixel_total : 18029 time to create 1 rle with old method : 0.021069049835205078 length of segment : 264 time for calcul the mask position with numpy : 0.0011668205261230469 nb_pixel_total : 75987 time to create 1 rle with old method : 0.08545207977294922 length of segment : 462 time for calcul the mask position with numpy : 0.0027952194213867188 nb_pixel_total : 104426 time to create 1 rle with old method : 0.11434316635131836 length of segment : 509 time for calcul the mask position with numpy : 0.0007712841033935547 nb_pixel_total : 15100 time to create 1 rle with old method : 0.016450881958007812 length of segment : 278 time for calcul the mask position with numpy : 0.00027632713317871094 nb_pixel_total : 7782 time to create 1 rle with old method : 0.009070873260498047 length of segment : 101 time for calcul the mask position with numpy : 0.001268625259399414 nb_pixel_total : 27181 time to create 1 rle with old method : 0.031584739685058594 length of segment : 341 time for calcul the mask position with numpy : 0.0006315708160400391 nb_pixel_total : 25634 time to create 1 rle with old method : 0.030138015747070312 length of segment : 174 time for calcul the mask position with numpy : 0.00013208389282226562 nb_pixel_total : 1777 time to create 1 rle with old method : 0.002214193344116211 length of segment : 58 time for calcul the mask position with numpy : 0.00016570091247558594 nb_pixel_total : 2025 time to create 1 rle with old method : 0.0025370121002197266 length of segment : 71 time for calcul the mask position with numpy : 0.0026786327362060547 nb_pixel_total : 92319 time to create 1 rle with old method : 0.10004925727844238 length of segment : 779 time for calcul the mask position with numpy : 0.001356363296508789 nb_pixel_total : 30945 time to create 1 rle with old method : 0.03561854362487793 length of segment : 367 time for calcul the mask position with numpy : 0.00017404556274414062 nb_pixel_total : 1799 time to create 1 rle with old method : 0.0023055076599121094 length of segment : 67 time for calcul the mask position with numpy : 0.0003032684326171875 nb_pixel_total : 9150 time to create 1 rle with old method : 0.010945558547973633 length of segment : 129 time for calcul the mask position with numpy : 0.00017452239990234375 nb_pixel_total : 3319 time to create 1 rle with old method : 0.004251241683959961 length of segment : 47 time for calcul the mask position with numpy : 0.0004029273986816406 nb_pixel_total : 8539 time to create 1 rle with old method : 0.010471343994140625 length of segment : 191 time for calcul the mask position with numpy : 0.002184629440307617 nb_pixel_total : 75186 time to create 1 rle with old method : 0.08586740493774414 length of segment : 440 time for calcul the mask position with numpy : 0.00025200843811035156 nb_pixel_total : 6312 time to create 1 rle with old method : 0.007865428924560547 length of segment : 57 time for calcul the mask position with numpy : 0.0016436576843261719 nb_pixel_total : 72815 time to create 1 rle with old method : 0.08365488052368164 length of segment : 330 time for calcul the mask position with numpy : 0.005937099456787109 nb_pixel_total : 213742 time to create 1 rle with new method : 0.012112140655517578 length of segment : 823 time for calcul the mask position with numpy : 0.0001723766326904297 nb_pixel_total : 1593 time to create 1 rle with old method : 0.0020895004272460938 length of segment : 41 time for calcul the mask position with numpy : 0.0035164356231689453 nb_pixel_total : 180961 time to create 1 rle with new method : 0.005143642425537109 length of segment : 424 time for calcul the mask position with numpy : 0.0009326934814453125 nb_pixel_total : 23927 time to create 1 rle with old method : 0.02734971046447754 length of segment : 340 time for calcul the mask position with numpy : 0.0009715557098388672 nb_pixel_total : 25335 time to create 1 rle with old method : 0.028997182846069336 length of segment : 268 time for calcul the mask position with numpy : 0.0006363391876220703 nb_pixel_total : 11341 time to create 1 rle with old method : 0.012976408004760742 length of segment : 200 time for calcul the mask position with numpy : 0.0021326541900634766 nb_pixel_total : 82573 time to create 1 rle with old method : 0.09243607521057129 length of segment : 366 time for calcul the mask position with numpy : 8.416175842285156e-05 nb_pixel_total : 1666 time to create 1 rle with old method : 0.00213623046875 length of segment : 41 time for calcul the mask position with numpy : 0.00020432472229003906 nb_pixel_total : 7572 time to create 1 rle with old method : 0.009473800659179688 length of segment : 120 time for calcul the mask position with numpy : 0.0003795623779296875 nb_pixel_total : 3978 time to create 1 rle with old method : 0.00509333610534668 length of segment : 140 time for calcul the mask position with numpy : 0.0003445148468017578 nb_pixel_total : 5160 time to create 1 rle with old method : 0.0070078372955322266 length of segment : 62 time for calcul the mask position with numpy : 0.00042891502380371094 nb_pixel_total : 11110 time to create 1 rle with old method : 0.013475656509399414 length of segment : 110 time for calcul the mask position with numpy : 0.00010848045349121094 nb_pixel_total : 1024 time to create 1 rle with old method : 0.001332998275756836 length of segment : 35 time for calcul the mask position with numpy : 0.0007121562957763672 nb_pixel_total : 22918 time to create 1 rle with old method : 0.02696537971496582 length of segment : 161 time for calcul the mask position with numpy : 0.0013687610626220703 nb_pixel_total : 20621 time to create 1 rle with old method : 0.023399829864501953 length of segment : 286 time for calcul the mask position with numpy : 0.0006780624389648438 nb_pixel_total : 14048 time to create 1 rle with old method : 0.01676321029663086 length of segment : 170 time for calcul the mask position with numpy : 0.0007457733154296875 nb_pixel_total : 10451 time to create 1 rle with old method : 0.01290583610534668 length of segment : 226 time for calcul the mask position with numpy : 0.00031375885009765625 nb_pixel_total : 5201 time to create 1 rle with old method : 0.0065882205963134766 length of segment : 78 time for calcul the mask position with numpy : 0.00018310546875 nb_pixel_total : 1942 time to create 1 rle with old method : 0.002567291259765625 length of segment : 55 time for calcul the mask position with numpy : 0.0019936561584472656 nb_pixel_total : 52316 time to create 1 rle with old method : 0.05913233757019043 length of segment : 357 time for calcul the mask position with numpy : 0.0007736682891845703 nb_pixel_total : 20461 time to create 1 rle with old method : 0.03310751914978027 length of segment : 179 time for calcul the mask position with numpy : 0.0011415481567382812 nb_pixel_total : 23634 time to create 1 rle with old method : 0.027385950088500977 length of segment : 303 time for calcul the mask position with numpy : 0.0002968311309814453 nb_pixel_total : 5097 time to create 1 rle with old method : 0.0066051483154296875 length of segment : 85 time for calcul the mask position with numpy : 0.00020885467529296875 nb_pixel_total : 2870 time to create 1 rle with old method : 0.003657102584838867 length of segment : 82 time for calcul the mask position with numpy : 0.00023746490478515625 nb_pixel_total : 2755 time to create 1 rle with old method : 0.003502368927001953 length of segment : 99 time for calcul the mask position with numpy : 0.0030717849731445312 nb_pixel_total : 82364 time to create 1 rle with old method : 0.09359598159790039 length of segment : 421 time for calcul the mask position with numpy : 0.000408172607421875 nb_pixel_total : 5866 time to create 1 rle with old method : 0.007487773895263672 length of segment : 119 time for calcul the mask position with numpy : 0.003999233245849609 nb_pixel_total : 150909 time to create 1 rle with new method : 0.007714509963989258 length of segment : 533 time for calcul the mask position with numpy : 0.0005986690521240234 nb_pixel_total : 22198 time to create 1 rle with old method : 0.026781320571899414 length of segment : 197 time for calcul the mask position with numpy : 0.0002391338348388672 nb_pixel_total : 6072 time to create 1 rle with old method : 0.008427619934082031 length of segment : 138 time for calcul the mask position with numpy : 0.00031638145446777344 nb_pixel_total : 10337 time to create 1 rle with old method : 0.013741731643676758 length of segment : 181 time spent for convertir_results : 16.8896484375 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 208 chid ids of type : 3663 Number RLEs to save : 59780 save missing photos in datou_result : time spend for datou_step_exec : 75.9033694267273 time spend to save output : 7.2471091747283936 total time spend for step 1 : 83.15047860145569 step2:crop_condition Fri Feb 7 01:35:53 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 208 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 11 About to insert : list_path_to_insert length 11 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 12 About to insert : list_path_to_insert length 12 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 11 About to insert : list_path_to_insert length 11 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 53 About to insert : list_path_to_insert length 53 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1335193060, 1335193055, 1335193052, 1335193049, 1335193011, 1335193003, 1335192999, 1335192939, 1335192938, 1335192936] Looping around the photos to save general results len do output : 99 /-3658502575Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502576Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502616Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502621Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502637Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502634Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502659Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502657Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502705Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502747Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502767Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502588Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502603Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502630Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502631Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502648Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502681Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502720Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502731Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502744Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502742Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502775Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502778Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502589Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502605Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502611Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502625Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502675Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502656Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502672Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502703Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502695Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502679Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502699Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502590Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502582Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502579Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502608Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502613Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502593Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502598Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502609Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502612Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502627Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502629Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502628Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502624Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502642Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502652Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502669Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502650Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502647Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502667Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502665Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502664Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502663Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502654Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502683Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502682Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502678Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502692Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502689Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502691Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502701Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502713Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502730Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502717Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502715Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502710Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502714Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502712Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502721Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502728Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502719Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502741Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502738Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502734Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502756Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502755Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502753Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502750Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502749Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502754Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502769Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502777Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502771Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502763Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502585Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502604Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502599Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502614Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502623Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502661Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502765Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502761Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502770Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502622Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502752Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658502596Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193060', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193055', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193052', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193049', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193011', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193003', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192999', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192939', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192938', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192936', None, None, None, None, None, '2560885') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 307 time used for this insertion : 0.03885173797607422 save_final save missing photos in datou_result : time spend for datou_step_exec : 48.80693602561951 time spend to save output : 0.04225659370422363 total time spend for step 2 : 48.84919261932373 step3:thcl Fri Feb 7 01:36:42 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.023589134216308594 time to convert the images to numpy array : 0.1973717212677002 time to import caffe and check if the image exist : 0.014347076416015625 time to convert the images to numpy array : 0.23915553092956543 time to import caffe and check if the image exist : 0.021764278411865234 time to convert the images to numpy array : 0.2443535327911377 time to import caffe and check if the image exist : 0.010946989059448242 time to convert the images to numpy array : 0.2667968273162842 time to import caffe and check if the image exist : 0.017473697662353516 time to convert the images to numpy array : 0.26761436462402344 time to import caffe and check if the image exist : 0.019123554229736328 time to convert the images to numpy array : 0.2714567184448242 time to import caffe and check if the image exist : 0.015035152435302734 time to convert the images to numpy array : 0.281383752822876 time to import caffe and check if the image exist : 0.019568204879760742 time to convert the images to numpy array : 0.3040120601654053 time to import caffe and check if the image exist : 0.015941619873046875 time to convert the images to numpy array : 0.31558799743652344 time to import caffe and check if the image exist : 0.01661205291748047 time to convert the images to numpy array : 0.3785686492919922 total time to convert the images to numpy array : 0.787407398223877 list photo_ids error: [] list photo_ids correct : [-3658502712, -3658502721, -3658502728, -3658502719, -3658502741, -3658502738, -3658502734, -3658502756, -3658502755, -3658502753, -3658502692, -3658502689, -3658502691, -3658502701, -3658502713, -3658502730, -3658502717, -3658502715, -3658502710, -3658502714, -3658502614, -3658502623, -3658502661, -3658502765, -3658502761, -3658502770, -3658502622, -3658502752, -3658502596, -3658502575, -3658502576, -3658502616, -3658502621, -3658502637, -3658502634, -3658502659, -3658502657, -3658502705, -3658502747, -3658502598, -3658502609, -3658502612, -3658502627, -3658502629, -3658502628, -3658502624, -3658502642, -3658502652, -3658502669, -3658502767, -3658502588, -3658502603, -3658502630, -3658502631, -3658502648, -3658502681, -3658502720, -3658502731, -3658502744, -3658502650, -3658502647, -3658502667, -3658502665, -3658502664, -3658502663, -3658502654, -3658502683, -3658502682, -3658502678, -3658502750, -3658502749, -3658502754, -3658502769, -3658502777, -3658502771, -3658502763, -3658502585, -3658502604, -3658502599, -3658502742, -3658502775, -3658502778, -3658502589, -3658502605, -3658502611, -3658502625, -3658502675, -3658502656, -3658502672, -3658502703, -3658502695, -3658502679, -3658502699, -3658502590, -3658502582, -3658502579, -3658502608, -3658502613, -3658502593] number of photos to traite : 99 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory havn't enough memory gpu , need / 2500 l 3632 free memory gpu now : 381 wait 20 seconds l 3637 free memory gpu now : 381 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5043 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.286773920059204 time used to do the prediction : 0.24745893478393555 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5041 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5041 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.4399476051330566 time used to do the prediction : 0.1748359203338623 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 2.2172927856445312e-05 save missing photos in datou_result : time spend for datou_step_exec : 30.834866762161255 time spend to save output : 0.0006721019744873047 total time spend for step 3 : 30.835538864135742 step4:merge_mask_thcl_custom Fri Feb 7 01:37:12 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 208 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1335193060, 1335193055, 1335193052, 1335193049, 1335193011, 1335193003, 1335192999, 1335192939, 1335192938, 1335192936] Looping around the photos to save general results len do output : 10 /1335193060Didn't retrieve data .Didn't retrieve data . /1335193055Didn't retrieve data .Didn't retrieve data . /1335193052Didn't retrieve data .Didn't retrieve data . /1335193049Didn't retrieve data .Didn't retrieve data . /1335193011Didn't retrieve data .Didn't retrieve data . /1335193003Didn't retrieve data .Didn't retrieve data . /1335192999Didn't retrieve data .Didn't retrieve data . /1335192939Didn't retrieve data .Didn't retrieve data . /1335192938Didn't retrieve data .Didn't retrieve data . /1335192936Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193060', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193055', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193052', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193049', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193011', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193003', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192999', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192939', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192938', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192936', None, None, None, None, None, '2560885') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.014281749725341797 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.06793928146362305 time spend to save output : 0.014770746231079102 total time spend for step 4 : 0.08271002769470215 step5:rle_unique_nms_with_priority Fri Feb 7 01:37:12 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 17 nb_hashtags : 6 time to prepare the origin masks : 9.301510095596313 time for calcul the mask position with numpy : 0.8863611221313477 nb_pixel_total : 4257345 time to create 1 rle with new method : 0.6973776817321777 time for calcul the mask position with numpy : 0.03199172019958496 nb_pixel_total : 17985 time to create 1 rle with old method : 0.020756244659423828 time for calcul the mask position with numpy : 0.03446078300476074 nb_pixel_total : 183153 time to create 1 rle with new method : 0.5335776805877686 time for calcul the mask position with numpy : 0.02852034568786621 nb_pixel_total : 32300 time to create 1 rle with old method : 0.039357900619506836 time for calcul the mask position with numpy : 0.03261995315551758 nb_pixel_total : 228418 time to create 1 rle with new method : 1.0653114318847656 time for calcul the mask position with numpy : 0.024457216262817383 nb_pixel_total : 86598 time to create 1 rle with old method : 0.10583972930908203 time for calcul the mask position with numpy : 0.024428129196166992 nb_pixel_total : 8457 time to create 1 rle with old method : 0.013609170913696289 time for calcul the mask position with numpy : 0.02213287353515625 nb_pixel_total : 1236 time to create 1 rle with old method : 0.0014767646789550781 time for calcul the mask position with numpy : 0.02236008644104004 nb_pixel_total : 68373 time to create 1 rle with old method : 0.08010339736938477 time for calcul the mask position with numpy : 0.02964019775390625 nb_pixel_total : 27407 time to create 1 rle with old method : 0.034148216247558594 time for calcul the mask position with numpy : 0.025819778442382812 nb_pixel_total : 150174 time to create 1 rle with new method : 0.9717276096343994 time for calcul the mask position with numpy : 0.028722524642944336 nb_pixel_total : 520612 time to create 1 rle with new method : 0.6054332256317139 time for calcul the mask position with numpy : 0.048340797424316406 nb_pixel_total : 1400100 time to create 1 rle with new method : 0.5357544422149658 time for calcul the mask position with numpy : 0.025395631790161133 nb_pixel_total : 11479 time to create 1 rle with old method : 0.01285552978515625 time for calcul the mask position with numpy : 0.022170543670654297 nb_pixel_total : 5830 time to create 1 rle with old method : 0.00711822509765625 time for calcul the mask position with numpy : 0.021203994750976562 nb_pixel_total : 35228 time to create 1 rle with old method : 0.03795671463012695 time for calcul the mask position with numpy : 0.020746231079101562 nb_pixel_total : 7304 time to create 1 rle with old method : 0.008071422576904297 time for calcul the mask position with numpy : 0.02276921272277832 nb_pixel_total : 8241 time to create 1 rle with old method : 0.009115219116210938 create new chi : 6.29490065574646 time to delete rle : 0.047121286392211914 batch 1 Loaded 18 chid ids of type : 3726 Number RLEs to save : 18645 TO DO : save crop sub photo not yet done ! save time : 2.966445207595825 nb_obj : 23 nb_hashtags : 6 time to prepare the origin masks : 10.237388372421265 time for calcul the mask position with numpy : 0.3669259548187256 nb_pixel_total : 5791558 time to create 1 rle with new method : 0.7984139919281006 time for calcul the mask position with numpy : 0.02335333824157715 nb_pixel_total : 947 time to create 1 rle with old method : 0.0015211105346679688 time for calcul the mask position with numpy : 0.022369384765625 nb_pixel_total : 24558 time to create 1 rle with old method : 0.027103185653686523 time for calcul the mask position with numpy : 0.02209615707397461 nb_pixel_total : 109315 time to create 1 rle with old method : 0.11922669410705566 time for calcul the mask position with numpy : 0.021129131317138672 nb_pixel_total : 18337 time to create 1 rle with old method : 0.02051997184753418 time for calcul the mask position with numpy : 0.02243185043334961 nb_pixel_total : 180429 time to create 1 rle with new method : 0.3974306583404541 time for calcul the mask position with numpy : 0.022652387619018555 nb_pixel_total : 16489 time to create 1 rle with old method : 0.026448726654052734 time for calcul the mask position with numpy : 0.035539865493774414 nb_pixel_total : 2094 time to create 1 rle with old method : 0.002463102340698242 time for calcul the mask position with numpy : 0.03468585014343262 nb_pixel_total : 3669 time to create 1 rle with old method : 0.004233360290527344 time for calcul the mask position with numpy : 0.03350377082824707 nb_pixel_total : 160854 time to create 1 rle with new method : 0.20377373695373535 time for calcul the mask position with numpy : 0.03578901290893555 nb_pixel_total : 4785 time to create 1 rle with old method : 0.0051212310791015625 time for calcul the mask position with numpy : 0.037386417388916016 nb_pixel_total : 4154 time to create 1 rle with old method : 0.004590034484863281 time for calcul the mask position with numpy : 0.04317140579223633 nb_pixel_total : 55088 time to create 1 rle with old method : 0.08536624908447266 time for calcul the mask position with numpy : 0.04143929481506348 nb_pixel_total : 14215 time to create 1 rle with old method : 0.01535797119140625 time for calcul the mask position with numpy : 0.034220218658447266 nb_pixel_total : 121210 time to create 1 rle with old method : 0.12975215911865234 time for calcul the mask position with numpy : 0.03715252876281738 nb_pixel_total : 44428 time to create 1 rle with old method : 0.046245574951171875 time for calcul the mask position with numpy : 0.03874993324279785 nb_pixel_total : 11350 time to create 1 rle with old method : 0.011851072311401367 time for calcul the mask position with numpy : 0.04466700553894043 nb_pixel_total : 258293 time to create 1 rle with new method : 0.3190467357635498 time for calcul the mask position with numpy : 0.03456568717956543 nb_pixel_total : 2370 time to create 1 rle with old method : 0.0027599334716796875 time for calcul the mask position with numpy : 0.040689706802368164 nb_pixel_total : 1418 time to create 1 rle with old method : 0.001588582992553711 time for calcul the mask position with numpy : 0.037877559661865234 nb_pixel_total : 6277 time to create 1 rle with old method : 0.006497621536254883 time for calcul the mask position with numpy : 0.03759002685546875 nb_pixel_total : 167175 time to create 1 rle with new method : 0.47907304763793945 time for calcul the mask position with numpy : 0.03784489631652832 nb_pixel_total : 45300 time to create 1 rle with old method : 0.04842257499694824 time for calcul the mask position with numpy : 0.03303098678588867 nb_pixel_total : 5927 time to create 1 rle with old method : 0.0067844390869140625 create new chi : 4.031267881393433 time to delete rle : 0.0022301673889160156 batch 1 Loaded 24 chid ids of type : 3726 Number RLEs to save : 14769 TO DO : save crop sub photo not yet done ! save time : 2.364715814590454 nb_obj : 20 nb_hashtags : 8 time to prepare the origin masks : 9.643543720245361 time for calcul the mask position with numpy : 0.2617173194885254 nb_pixel_total : 5619781 time to create 1 rle with new method : 1.0657286643981934 time for calcul the mask position with numpy : 0.022935152053833008 nb_pixel_total : 128607 time to create 1 rle with old method : 0.14045381546020508 time for calcul the mask position with numpy : 0.022211074829101562 nb_pixel_total : 77779 time to create 1 rle with old method : 0.0873880386352539 time for calcul the mask position with numpy : 0.021642446517944336 nb_pixel_total : 7252 time to create 1 rle with old method : 0.008700132369995117 time for calcul the mask position with numpy : 0.02207016944885254 nb_pixel_total : 1653 time to create 1 rle with old method : 0.0019834041595458984 time for calcul the mask position with numpy : 0.021497488021850586 nb_pixel_total : 1055 time to create 1 rle with old method : 0.0015125274658203125 time for calcul the mask position with numpy : 0.025460243225097656 nb_pixel_total : 312537 time to create 1 rle with new method : 0.38855910301208496 time for calcul the mask position with numpy : 0.024296998977661133 nb_pixel_total : 160307 time to create 1 rle with new method : 0.5807821750640869 time for calcul the mask position with numpy : 0.021868228912353516 nb_pixel_total : 47926 time to create 1 rle with old method : 0.06287789344787598 time for calcul the mask position with numpy : 0.0355679988861084 nb_pixel_total : 4415 time to create 1 rle with old method : 0.005277872085571289 time for calcul the mask position with numpy : 0.023023605346679688 nb_pixel_total : 11413 time to create 1 rle with old method : 0.014448404312133789 time for calcul the mask position with numpy : 0.022557735443115234 nb_pixel_total : 43253 time to create 1 rle with old method : 0.04924154281616211 time for calcul the mask position with numpy : 0.024506568908691406 nb_pixel_total : 192864 time to create 1 rle with new method : 0.2536473274230957 time for calcul the mask position with numpy : 0.02499675750732422 nb_pixel_total : 43887 time to create 1 rle with old method : 0.04911208152770996 time for calcul the mask position with numpy : 0.025148868560791016 nb_pixel_total : 5666 time to create 1 rle with old method : 0.0065877437591552734 time for calcul the mask position with numpy : 0.025176286697387695 nb_pixel_total : 112160 time to create 1 rle with old method : 0.1581737995147705 time for calcul the mask position with numpy : 0.023920536041259766 nb_pixel_total : 29561 time to create 1 rle with old method : 0.039255619049072266 time for calcul the mask position with numpy : 0.040270090103149414 nb_pixel_total : 3323 time to create 1 rle with old method : 0.0038366317749023438 time for calcul the mask position with numpy : 0.03996443748474121 nb_pixel_total : 66267 time to create 1 rle with old method : 0.07203292846679688 time for calcul the mask position with numpy : 0.039235591888427734 nb_pixel_total : 5796 time to create 1 rle with old method : 0.006591320037841797 time for calcul the mask position with numpy : 0.04499411582946777 nb_pixel_total : 174738 time to create 1 rle with new method : 0.4277610778808594 create new chi : 4.359973192214966 time to delete rle : 0.0015637874603271484 batch 1 Loaded 21 chid ids of type : 3726 Number RLEs to save : 13959 TO DO : save crop sub photo not yet done ! save time : 2.1851344108581543 nb_obj : 12 nb_hashtags : 3 time to prepare the origin masks : 4.132434129714966 time for calcul the mask position with numpy : 0.14110922813415527 nb_pixel_total : 4767699 time to create 1 rle with new method : 0.7013585567474365 time for calcul the mask position with numpy : 0.021780729293823242 nb_pixel_total : 3082 time to create 1 rle with old method : 0.003535747528076172 time for calcul the mask position with numpy : 0.021737098693847656 nb_pixel_total : 7798 time to create 1 rle with old method : 0.009020090103149414 time for calcul the mask position with numpy : 0.021854400634765625 nb_pixel_total : 73209 time to create 1 rle with old method : 0.08197474479675293 time for calcul the mask position with numpy : 0.021480798721313477 nb_pixel_total : 2783 time to create 1 rle with old method : 0.0032546520233154297 time for calcul the mask position with numpy : 0.023802518844604492 nb_pixel_total : 3974 time to create 1 rle with old method : 0.007359027862548828 time for calcul the mask position with numpy : 0.0287628173828125 nb_pixel_total : 748418 time to create 1 rle with new method : 0.30318784713745117 time for calcul the mask position with numpy : 0.020472288131713867 nb_pixel_total : 4712 time to create 1 rle with old method : 0.005339622497558594 time for calcul the mask position with numpy : 0.02378082275390625 nb_pixel_total : 58443 time to create 1 rle with old method : 0.06280660629272461 time for calcul the mask position with numpy : 0.02089095115661621 nb_pixel_total : 2573 time to create 1 rle with old method : 0.0028772354125976562 time for calcul the mask position with numpy : 0.022234201431274414 nb_pixel_total : 22427 time to create 1 rle with old method : 0.024899959564208984 time for calcul the mask position with numpy : 0.04868030548095703 nb_pixel_total : 1350390 time to create 1 rle with new method : 0.3120129108428955 time for calcul the mask position with numpy : 0.04737210273742676 nb_pixel_total : 4732 time to create 1 rle with old method : 0.006323814392089844 create new chi : 2.063680410385132 time to delete rle : 0.001211404800415039 batch 1 Loaded 13 chid ids of type : 3726 Number RLEs to save : 11189 TO DO : save crop sub photo not yet done ! save time : 1.733031988143921 nb_obj : 28 nb_hashtags : 5 time to prepare the origin masks : 10.142802715301514 time for calcul the mask position with numpy : 0.07845020294189453 nb_pixel_total : 5977260 time to create 1 rle with new method : 0.14765143394470215 time for calcul the mask position with numpy : 0.029805898666381836 nb_pixel_total : 56141 time to create 1 rle with old method : 0.05962491035461426 time for calcul the mask position with numpy : 0.027374267578125 nb_pixel_total : 6663 time to create 1 rle with old method : 0.0069544315338134766 time for calcul the mask position with numpy : 0.029285669326782227 nb_pixel_total : 10002 time to create 1 rle with old method : 0.015796422958374023 time for calcul the mask position with numpy : 0.033037662506103516 nb_pixel_total : 19938 time to create 1 rle with old method : 0.027947425842285156 time for calcul the mask position with numpy : 0.0274658203125 nb_pixel_total : 20023 time to create 1 rle with old method : 0.021422863006591797 time for calcul the mask position with numpy : 0.02889847755432129 nb_pixel_total : 26551 time to create 1 rle with old method : 0.029187917709350586 time for calcul the mask position with numpy : 0.028134822845458984 nb_pixel_total : 9039 time to create 1 rle with old method : 0.009969472885131836 time for calcul the mask position with numpy : 0.030711889266967773 nb_pixel_total : 202353 time to create 1 rle with new method : 0.12683486938476562 time for calcul the mask position with numpy : 0.028548002243041992 nb_pixel_total : 5329 time to create 1 rle with old method : 0.006050586700439453 time for calcul the mask position with numpy : 0.031191349029541016 nb_pixel_total : 64944 time to create 1 rle with old method : 0.06999421119689941 time for calcul the mask position with numpy : 0.02815532684326172 nb_pixel_total : 2015 time to create 1 rle with old method : 0.002556324005126953 time for calcul the mask position with numpy : 0.02788829803466797 nb_pixel_total : 8763 time to create 1 rle with old method : 0.009785890579223633 time for calcul the mask position with numpy : 0.028616905212402344 nb_pixel_total : 48791 time to create 1 rle with old method : 0.052696943283081055 time for calcul the mask position with numpy : 0.029122114181518555 nb_pixel_total : 58159 time to create 1 rle with old method : 0.06357431411743164 time for calcul the mask position with numpy : 0.030392169952392578 nb_pixel_total : 157718 time to create 1 rle with new method : 0.13193845748901367 time for calcul the mask position with numpy : 0.029717683792114258 nb_pixel_total : 58675 time to create 1 rle with old method : 0.0692903995513916 time for calcul the mask position with numpy : 0.03154468536376953 nb_pixel_total : 11420 time to create 1 rle with old method : 0.013102054595947266 time for calcul the mask position with numpy : 0.03016805648803711 nb_pixel_total : 73876 time to create 1 rle with old method : 0.08414220809936523 time for calcul the mask position with numpy : 0.029466867446899414 nb_pixel_total : 13371 time to create 1 rle with old method : 0.015262126922607422 time for calcul the mask position with numpy : 0.0295257568359375 nb_pixel_total : 28774 time to create 1 rle with old method : 0.03807568550109863 time for calcul the mask position with numpy : 0.034723758697509766 nb_pixel_total : 11794 time to create 1 rle with old method : 0.015598535537719727 time for calcul the mask position with numpy : 0.03348708152770996 nb_pixel_total : 83063 time to create 1 rle with old method : 0.1002655029296875 time for calcul the mask position with numpy : 0.030121326446533203 nb_pixel_total : 1469 time to create 1 rle with old method : 0.0019500255584716797 time for calcul the mask position with numpy : 0.05331087112426758 nb_pixel_total : 8934 time to create 1 rle with old method : 0.04529714584350586 time for calcul the mask position with numpy : 0.09522247314453125 nb_pixel_total : 73539 time to create 1 rle with old method : 0.15401721000671387 time for calcul the mask position with numpy : 0.07799482345581055 nb_pixel_total : 1524 time to create 1 rle with old method : 0.002804279327392578 time for calcul the mask position with numpy : 0.0379023551940918 nb_pixel_total : 5725 time to create 1 rle with old method : 0.010454893112182617 time for calcul the mask position with numpy : 0.03834271430969238 nb_pixel_total : 4387 time to create 1 rle with old method : 0.008129119873046875 create new chi : 2.4537100791931152 time to delete rle : 0.0033228397369384766 batch 1 Loaded 29 chid ids of type : 3726 Number RLEs to save : 15520 TO DO : save crop sub photo not yet done ! save time : 1.6766033172607422 nb_obj : 25 nb_hashtags : 5 time to prepare the origin masks : 12.705338478088379 time for calcul the mask position with numpy : 0.17855072021484375 nb_pixel_total : 6072402 time to create 1 rle with new method : 0.6905145645141602 time for calcul the mask position with numpy : 0.02063274383544922 nb_pixel_total : 79239 time to create 1 rle with old method : 0.0857391357421875 time for calcul the mask position with numpy : 0.023610591888427734 nb_pixel_total : 53677 time to create 1 rle with old method : 0.055196285247802734 time for calcul the mask position with numpy : 0.01992964744567871 nb_pixel_total : 21585 time to create 1 rle with old method : 0.02283477783203125 time for calcul the mask position with numpy : 0.019616365432739258 nb_pixel_total : 1497 time to create 1 rle with old method : 0.0016620159149169922 time for calcul the mask position with numpy : 0.019943714141845703 nb_pixel_total : 4788 time to create 1 rle with old method : 0.005381345748901367 time for calcul the mask position with numpy : 0.020042896270751953 nb_pixel_total : 7086 time to create 1 rle with old method : 0.007912874221801758 time for calcul the mask position with numpy : 0.020467042922973633 nb_pixel_total : 2093 time to create 1 rle with old method : 0.0023076534271240234 time for calcul the mask position with numpy : 0.021832942962646484 nb_pixel_total : 154666 time to create 1 rle with new method : 0.513573408126831 time for calcul the mask position with numpy : 0.01972794532775879 nb_pixel_total : 5400 time to create 1 rle with old method : 0.005820751190185547 time for calcul the mask position with numpy : 0.020492076873779297 nb_pixel_total : 22764 time to create 1 rle with old method : 0.024938344955444336 time for calcul the mask position with numpy : 0.022189617156982422 nb_pixel_total : 11592 time to create 1 rle with old method : 0.012182474136352539 time for calcul the mask position with numpy : 0.025181293487548828 nb_pixel_total : 68964 time to create 1 rle with old method : 0.0718071460723877 time for calcul the mask position with numpy : 0.021157264709472656 nb_pixel_total : 45703 time to create 1 rle with old method : 0.04734086990356445 time for calcul the mask position with numpy : 0.021638154983520508 nb_pixel_total : 1541 time to create 1 rle with old method : 0.001857757568359375 time for calcul the mask position with numpy : 0.0219423770904541 nb_pixel_total : 25624 time to create 1 rle with old method : 0.027983665466308594 time for calcul the mask position with numpy : 0.02159404754638672 nb_pixel_total : 83197 time to create 1 rle with old method : 0.08763718605041504 time for calcul the mask position with numpy : 0.02023601531982422 nb_pixel_total : 5185 time to create 1 rle with old method : 0.006167173385620117 time for calcul the mask position with numpy : 0.021698951721191406 nb_pixel_total : 1209 time to create 1 rle with old method : 0.0013544559478759766 time for calcul the mask position with numpy : 0.022411346435546875 nb_pixel_total : 201630 time to create 1 rle with new method : 0.8914978504180908 time for calcul the mask position with numpy : 0.03473496437072754 nb_pixel_total : 76363 time to create 1 rle with old method : 0.08242368698120117 time for calcul the mask position with numpy : 0.034543752670288086 nb_pixel_total : 28096 time to create 1 rle with old method : 0.032715797424316406 time for calcul the mask position with numpy : 0.034330129623413086 nb_pixel_total : 3321 time to create 1 rle with old method : 0.0037899017333984375 time for calcul the mask position with numpy : 0.036407470703125 nb_pixel_total : 11443 time to create 1 rle with old method : 0.012775897979736328 time for calcul the mask position with numpy : 0.033097028732299805 nb_pixel_total : 10799 time to create 1 rle with old method : 0.01214289665222168 time for calcul the mask position with numpy : 0.03188371658325195 nb_pixel_total : 50376 time to create 1 rle with old method : 0.0548551082611084 create new chi : 3.6234352588653564 time to delete rle : 0.0012683868408203125 batch 1 Loaded 26 chid ids of type : 3726 Number RLEs to save : 14031 TO DO : save crop sub photo not yet done ! save time : 1.3007700443267822 nb_obj : 26 nb_hashtags : 7 time to prepare the origin masks : 8.92792010307312 time for calcul the mask position with numpy : 0.09103250503540039 nb_pixel_total : 6291932 time to create 1 rle with new method : 0.15542221069335938 time for calcul the mask position with numpy : 0.02676534652709961 nb_pixel_total : 1326 time to create 1 rle with old method : 0.0014801025390625 time for calcul the mask position with numpy : 0.02735757827758789 nb_pixel_total : 162591 time to create 1 rle with new method : 0.13924145698547363 time for calcul the mask position with numpy : 0.026459693908691406 nb_pixel_total : 2469 time to create 1 rle with old method : 0.0025475025177001953 time for calcul the mask position with numpy : 0.026912927627563477 nb_pixel_total : 106695 time to create 1 rle with old method : 0.10914087295532227 time for calcul the mask position with numpy : 0.027198314666748047 nb_pixel_total : 41007 time to create 1 rle with old method : 0.04296231269836426 time for calcul the mask position with numpy : 0.026952505111694336 nb_pixel_total : 5216 time to create 1 rle with old method : 0.008335113525390625 time for calcul the mask position with numpy : 0.03319144248962402 nb_pixel_total : 78308 time to create 1 rle with old method : 0.0938115119934082 time for calcul the mask position with numpy : 0.027956247329711914 nb_pixel_total : 5191 time to create 1 rle with old method : 0.005944967269897461 time for calcul the mask position with numpy : 0.028072118759155273 nb_pixel_total : 90658 time to create 1 rle with old method : 0.09794831275939941 time for calcul the mask position with numpy : 0.029378175735473633 nb_pixel_total : 11394 time to create 1 rle with old method : 0.012637853622436523 time for calcul the mask position with numpy : 0.03104400634765625 nb_pixel_total : 2222 time to create 1 rle with old method : 0.003633260726928711 time for calcul the mask position with numpy : 0.03229117393493652 nb_pixel_total : 58638 time to create 1 rle with old method : 0.06422233581542969 time for calcul the mask position with numpy : 0.029470205307006836 nb_pixel_total : 575 time to create 1 rle with old method : 0.0008342266082763672 time for calcul the mask position with numpy : 0.029771089553833008 nb_pixel_total : 46705 time to create 1 rle with old method : 0.05142927169799805 time for calcul the mask position with numpy : 0.02931976318359375 nb_pixel_total : 2561 time to create 1 rle with old method : 0.0030024051666259766 time for calcul the mask position with numpy : 0.029129981994628906 nb_pixel_total : 10884 time to create 1 rle with old method : 0.012681007385253906 time for calcul the mask position with numpy : 0.02915215492248535 nb_pixel_total : 1826 time to create 1 rle with old method : 0.0021791458129882812 time for calcul the mask position with numpy : 0.029375314712524414 nb_pixel_total : 41305 time to create 1 rle with old method : 0.045144081115722656 time for calcul the mask position with numpy : 0.029330015182495117 nb_pixel_total : 5402 time to create 1 rle with old method : 0.006352424621582031 time for calcul the mask position with numpy : 0.029354095458984375 nb_pixel_total : 12182 time to create 1 rle with old method : 0.014685392379760742 time for calcul the mask position with numpy : 0.029347658157348633 nb_pixel_total : 20269 time to create 1 rle with old method : 0.022437572479248047 time for calcul the mask position with numpy : 0.029297828674316406 nb_pixel_total : 5767 time to create 1 rle with old method : 0.0066111087799072266 time for calcul the mask position with numpy : 0.029735326766967773 nb_pixel_total : 15930 time to create 1 rle with old method : 0.02732229232788086 time for calcul the mask position with numpy : 0.033013343811035156 nb_pixel_total : 19471 time to create 1 rle with old method : 0.021468639373779297 time for calcul the mask position with numpy : 0.029340267181396484 nb_pixel_total : 3022 time to create 1 rle with old method : 0.0035467147827148438 time for calcul the mask position with numpy : 0.029381752014160156 nb_pixel_total : 6694 time to create 1 rle with old method : 0.007586240768432617 create new chi : 1.8461694717407227 time to delete rle : 0.0011813640594482422 batch 1 Loaded 27 chid ids of type : 3726 Number RLEs to save : 12410 TO DO : save crop sub photo not yet done ! save time : 1.3568861484527588 nb_obj : 13 nb_hashtags : 4 time to prepare the origin masks : 3.776693344116211 time for calcul the mask position with numpy : 0.35857510566711426 nb_pixel_total : 6720634 time to create 1 rle with new method : 0.9868462085723877 time for calcul the mask position with numpy : 0.023175954818725586 nb_pixel_total : 104423 time to create 1 rle with old method : 0.10779833793640137 time for calcul the mask position with numpy : 0.02284693717956543 nb_pixel_total : 8520 time to create 1 rle with old method : 0.009407281875610352 time for calcul the mask position with numpy : 0.023842334747314453 nb_pixel_total : 30779 time to create 1 rle with old method : 0.03258800506591797 time for calcul the mask position with numpy : 0.02283930778503418 nb_pixel_total : 3311 time to create 1 rle with old method : 0.003481149673461914 time for calcul the mask position with numpy : 0.02199387550354004 nb_pixel_total : 1787 time to create 1 rle with old method : 0.0020427703857421875 time for calcul the mask position with numpy : 0.023470640182495117 nb_pixel_total : 2002 time to create 1 rle with old method : 0.002246856689453125 time for calcul the mask position with numpy : 0.022087574005126953 nb_pixel_total : 1768 time to create 1 rle with old method : 0.0019307136535644531 time for calcul the mask position with numpy : 0.023082494735717773 nb_pixel_total : 27085 time to create 1 rle with old method : 0.028838634490966797 time for calcul the mask position with numpy : 0.022240877151489258 nb_pixel_total : 7763 time to create 1 rle with old method : 0.008517265319824219 time for calcul the mask position with numpy : 0.02269768714904785 nb_pixel_total : 9117 time to create 1 rle with old method : 0.009919881820678711 time for calcul the mask position with numpy : 0.02257847785949707 nb_pixel_total : 14950 time to create 1 rle with old method : 0.015557050704956055 time for calcul the mask position with numpy : 0.023063182830810547 nb_pixel_total : 25509 time to create 1 rle with old method : 0.027157306671142578 time for calcul the mask position with numpy : 0.02455878257751465 nb_pixel_total : 92592 time to create 1 rle with old method : 0.0965883731842041 create new chi : 2.0216076374053955 time to delete rle : 0.0007092952728271484 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 8302 TO DO : save crop sub photo not yet done ! save time : 1.4717998504638672 nb_obj : 14 nb_hashtags : 8 time to prepare the origin masks : 6.482939720153809 time for calcul the mask position with numpy : 0.19684576988220215 nb_pixel_total : 6340255 time to create 1 rle with new method : 1.1308341026306152 time for calcul the mask position with numpy : 0.024380922317504883 nb_pixel_total : 23891 time to create 1 rle with old method : 0.026383161544799805 time for calcul the mask position with numpy : 0.023357629776000977 nb_pixel_total : 82399 time to create 1 rle with old method : 0.0905447006225586 time for calcul the mask position with numpy : 0.024101972579956055 nb_pixel_total : 11309 time to create 1 rle with old method : 0.01245880126953125 time for calcul the mask position with numpy : 0.02306652069091797 nb_pixel_total : 5026 time to create 1 rle with old method : 0.005792140960693359 time for calcul the mask position with numpy : 0.024364709854125977 nb_pixel_total : 212654 time to create 1 rle with new method : 0.1906750202178955 time for calcul the mask position with numpy : 0.0232086181640625 nb_pixel_total : 3962 time to create 1 rle with old method : 0.004616260528564453 time for calcul the mask position with numpy : 0.024799108505249023 nb_pixel_total : 7537 time to create 1 rle with old method : 0.008812189102172852 time for calcul the mask position with numpy : 0.0250546932220459 nb_pixel_total : 1661 time to create 1 rle with old method : 0.0020673274993896484 time for calcul the mask position with numpy : 0.02336263656616211 nb_pixel_total : 25300 time to create 1 rle with old method : 0.02837848663330078 time for calcul the mask position with numpy : 0.024735689163208008 nb_pixel_total : 1586 time to create 1 rle with old method : 0.0019190311431884766 time for calcul the mask position with numpy : 0.02355337142944336 nb_pixel_total : 6281 time to create 1 rle with old method : 0.007382392883300781 time for calcul the mask position with numpy : 0.025313854217529297 nb_pixel_total : 75064 time to create 1 rle with old method : 0.08215594291687012 time for calcul the mask position with numpy : 0.025672435760498047 nb_pixel_total : 180767 time to create 1 rle with new method : 0.19423460960388184 time for calcul the mask position with numpy : 0.027851581573486328 nb_pixel_total : 72548 time to create 1 rle with old method : 0.07947087287902832 create new chi : 2.477126359939575 time to delete rle : 0.0009789466857910156 batch 1 Loaded 15 chid ids of type : 3726 Number RLEs to save : 9416 TO DO : save crop sub photo not yet done ! save time : 1.2433874607086182 nb_obj : 19 nb_hashtags : 6 time to prepare the origin masks : 7.80269455909729 time for calcul the mask position with numpy : 0.18072724342346191 nb_pixel_total : 6614859 time to create 1 rle with new method : 1.0117008686065674 time for calcul the mask position with numpy : 0.024860620498657227 nb_pixel_total : 82115 time to create 1 rle with old method : 0.084259033203125 time for calcul the mask position with numpy : 0.02193617820739746 nb_pixel_total : 1591 time to create 1 rle with old method : 0.0017354488372802734 time for calcul the mask position with numpy : 0.022287368774414062 nb_pixel_total : 5845 time to create 1 rle with old method : 0.0061833858489990234 time for calcul the mask position with numpy : 0.021931171417236328 nb_pixel_total : 51798 time to create 1 rle with old method : 0.05339241027832031 time for calcul the mask position with numpy : 0.02334451675415039 nb_pixel_total : 150759 time to create 1 rle with new method : 0.7504022121429443 time for calcul the mask position with numpy : 0.027227401733398438 nb_pixel_total : 5174 time to create 1 rle with old method : 0.006001949310302734 time for calcul the mask position with numpy : 0.025400400161743164 nb_pixel_total : 2737 time to create 1 rle with old method : 0.003073453903198242 time for calcul the mask position with numpy : 0.029382944107055664 nb_pixel_total : 2858 time to create 1 rle with old method : 0.003251791000366211 time for calcul the mask position with numpy : 0.029020071029663086 nb_pixel_total : 5063 time to create 1 rle with old method : 0.00579524040222168 time for calcul the mask position with numpy : 0.029813051223754883 nb_pixel_total : 28 time to create 1 rle with old method : 0.0002048015594482422 time for calcul the mask position with numpy : 0.03143429756164551 nb_pixel_total : 1928 time to create 1 rle with old method : 0.002295255661010742 time for calcul the mask position with numpy : 0.041153669357299805 nb_pixel_total : 23588 time to create 1 rle with old method : 0.026410818099975586 time for calcul the mask position with numpy : 0.030604839324951172 nb_pixel_total : 22158 time to create 1 rle with old method : 0.024378299713134766 time for calcul the mask position with numpy : 0.0262448787689209 nb_pixel_total : 22835 time to create 1 rle with old method : 0.025429725646972656 time for calcul the mask position with numpy : 0.026720285415649414 nb_pixel_total : 10355 time to create 1 rle with old method : 0.016951799392700195 time for calcul the mask position with numpy : 0.030459165573120117 nb_pixel_total : 11074 time to create 1 rle with old method : 0.0196073055267334 time for calcul the mask position with numpy : 0.03548383712768555 nb_pixel_total : 20473 time to create 1 rle with old method : 0.022628068923950195 time for calcul the mask position with numpy : 0.03001880645751953 nb_pixel_total : 1018 time to create 1 rle with old method : 0.0012679100036621094 time for calcul the mask position with numpy : 0.028238296508789062 nb_pixel_total : 13984 time to create 1 rle with old method : 0.016063213348388672 create new chi : 2.8630523681640625 time to delete rle : 0.0009624958038330078 batch 1 Loaded 20 chid ids of type : 3726 Number RLEs to save : 8931 TO DO : save crop sub photo not yet done ! save time : 1.384068489074707 map_output_result : {1335193060: (0.0, 'Should be the crop_list due to order', 0.0), 1335193055: (0.0, 'Should be the crop_list due to order', 0.0), 1335193052: (0.0, 'Should be the crop_list due to order', 0.0), 1335193049: (0.0, 'Should be the crop_list due to order', 0.0), 1335193011: (0.0, 'Should be the crop_list due to order', 0.0), 1335193003: (0.0, 'Should be the crop_list due to order', 0.0), 1335192999: (0.0, 'Should be the crop_list due to order', 0.0), 1335192939: (0.0, 'Should be the crop_list due to order', 0.0), 1335192938: (0.0, 'Should be the crop_list due to order', 0.0), 1335192936: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1335193060, 1335193055, 1335193052, 1335193049, 1335193011, 1335193003, 1335192999, 1335192939, 1335192938, 1335192936] Looping around the photos to save general results len do output : 10 /1335193060.Didn't retrieve data . /1335193055.Didn't retrieve data . /1335193052.Didn't retrieve data . /1335193049.Didn't retrieve data . /1335193011.Didn't retrieve data . /1335193003.Didn't retrieve data . /1335192999.Didn't retrieve data . /1335192939.Didn't retrieve data . /1335192938.Didn't retrieve data . /1335192936.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193060', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193055', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193052', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193049', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193011', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193003', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192999', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192939', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192938', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192936', None, None, None, None, None, '2560885') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.01656317710876465 save_final save missing photos in datou_result : time spend for datou_step_exec : 134.16905117034912 time spend to save output : 0.018368005752563477 total time spend for step 5 : 134.18741917610168 step6:crop_condition Fri Feb 7 01:39:27 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 207 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! About to upload 7 photos upload in portfolio : 4869462 init cache_photo without model_param we have 7 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738888770_929578 we have uploaded 7 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.1853857040405273 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738888774_929578 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.9107949733734131 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! About to upload 5 photos upload in portfolio : 4869462 init cache_photo without model_param we have 5 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738888778_929578 we have uploaded 5 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.3496532440185547 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 8 About to insert : list_path_to_insert length 8 new photo from crops ! About to upload 8 photos upload in portfolio : 4869462 init cache_photo without model_param we have 8 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738888784_929578 we have uploaded 8 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.3534555435180664 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738888788_929578 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.1465699672698975 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738888794_929578 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.4027643203735352 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 Next one ! map_result returned by crop_photo_return_map_crop : length : 1 About to insert : list_path_to_insert length 1 new photo from crops ! About to upload 1 photos upload in portfolio : 4869462 init cache_photo without model_param we have 1 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738888796_929578 we have uploaded 1 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.7560586929321289 we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1335193060, 1335193055, 1335193052, 1335193049, 1335193011, 1335193003, 1335192999, 1335192939, 1335192938, 1335192936] Looping around the photos to save general results len do output : 32 /1335332231Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332232Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332233Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332234Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332235Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332236Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332237Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332241Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332243Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332244Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332264Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332266Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332268Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332270Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332272Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332325Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332326Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332327Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332328Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332329Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332330Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332331Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332332Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332335Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332336Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332337Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332338Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332342Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332343Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332344Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332345Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335332346Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193060', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193055', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193052', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193049', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193011', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193003', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192999', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192939', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192938', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192936', None, None, None, None, None, '2560885') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 106 time used for this insertion : 0.03966569900512695 save_final save missing photos in datou_result : time spend for datou_step_exec : 33.638219356536865 time spend to save output : 0.040654897689819336 total time spend for step 6 : 33.678874254226685 step7:ventilate_hashtags_in_portfolio Fri Feb 7 01:40:00 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20314827 get user id for portfolio 20314827 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314827 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('environnement','Teint_Dans_La_Masse','flou','kraft','metal','mal_croppe','Carton_gris','papier','Carton_brun','autre_refus','cartonnette','plastique')) AND mptpi.`min_score`=0.5 To do To do SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314827 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('environnement','Teint_Dans_La_Masse','flou','kraft','metal','mal_croppe','Carton_gris','papier','Carton_brun','autre_refus','cartonnette','plastique')) AND mptpi.`min_score`=0.5 To do To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314827 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('environnement','Teint_Dans_La_Masse','flou','kraft','metal','mal_croppe','Carton_gris','papier','Carton_brun','autre_refus','cartonnette','plastique')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20314926,20314927,20314928,20314929,20314930,20314931,20314932,20314933,20314934,20314935,20314936,20314937?tags=environnement,Teint_Dans_La_Masse,flou,kraft,metal,mal_croppe,Carton_gris,papier,Carton_brun,autre_refus,cartonnette,plastique Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1335193060, 1335193055, 1335193052, 1335193049, 1335193011, 1335193003, 1335192999, 1335192939, 1335192938, 1335192936] Looping around the photos to save general results len do output : 1 /20314827. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193060', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193055', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193052', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193049', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193011', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193003', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192999', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192939', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192938', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192936', None, None, None, None, None, '2560885') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.19898533821105957 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.7720012664794922 time spend to save output : 0.19930291175842285 total time spend for step 7 : 0.971304178237915 step8:final Fri Feb 7 01:40:01 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1335193060: ('0.07702728454025964',), 1335193055: ('0.07702728454025964',), 1335193052: ('0.07702728454025964',), 1335193049: ('0.07702728454025964',), 1335193011: ('0.07702728454025964',), 1335193003: ('0.07702728454025964',), 1335192999: ('0.07702728454025964',), 1335192939: ('0.07702728454025964',), 1335192938: ('0.07702728454025964',), 1335192936: ('0.07702728454025964',)} new output for save of step final : {1335193060: ('0.07702728454025964',), 1335193055: ('0.07702728454025964',), 1335193052: ('0.07702728454025964',), 1335193049: ('0.07702728454025964',), 1335193011: ('0.07702728454025964',), 1335193003: ('0.07702728454025964',), 1335192999: ('0.07702728454025964',), 1335192939: ('0.07702728454025964',), 1335192938: ('0.07702728454025964',), 1335192936: ('0.07702728454025964',)} [1335193060, 1335193055, 1335193052, 1335193049, 1335193011, 1335193003, 1335192999, 1335192939, 1335192938, 1335192936] Looping around the photos to save general results len do output : 10 /1335193060.Didn't retrieve data . /1335193055.Didn't retrieve data . /1335193052.Didn't retrieve data . /1335193049.Didn't retrieve data . /1335193011.Didn't retrieve data . /1335193003.Didn't retrieve data . /1335192999.Didn't retrieve data . /1335192939.Didn't retrieve data . /1335192938.Didn't retrieve data . /1335192936.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193060', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193055', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193052', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193049', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193011', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193003', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192999', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192939', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192938', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192936', None, None, None, None, None, '2560885') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.39476990699768066 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.22355365753173828 time spend to save output : 0.3959331512451172 total time spend for step 8 : 0.6194868087768555 step9:velours_tree Fri Feb 7 01:40:02 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.10342645645141602 time spend to save output : 3.814697265625e-05 total time spend for step 9 : 0.10346460342407227 step10:send_mail_cod Fri Feb 7 01:40:02 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20314827_07-02-2025_01_40_02.pdf 20314928 imagette203149281738888802 20314929 change filename to text .imagette203149291738888802 20314930 imagette203149301738888802 20314931 imagette203149311738888802 20314935 change filename to text .change filename to text .change filename to text .imagette203149351738888802 20314936 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203149361738888802 20314937 change filename to text .change filename to text .change filename to text .change filename to text .imagette203149371738888803 20314927 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203149271738888803 20314932 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203149321738888804 20314934 change filename to text .change filename to text .change filename to text .change filename to text .imagette203149341738888804 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20314827 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20314926,20314927,20314928,20314929,20314930,20314931,20314932,20314933,20314934,20314935,20314936,20314937?tags=environnement,Teint_Dans_La_Masse,flou,kraft,metal,mal_croppe,Carton_gris,papier,Carton_brun,autre_refus,cartonnette,plastique your option no_mail is active, we will not send the real mail to your client args[1335193060] : ('0.07702728454025964',) no score found for photo 1335193060 apple ('0.07702728454025964',) We are sending mail with results at report@fotonower.com args[1335193055] : ('0.07702728454025964',) no score found for photo 1335193055 apple ('0.07702728454025964',) We are sending mail with results at report@fotonower.com args[1335193052] : ('0.07702728454025964',) no score found for photo 1335193052 apple ('0.07702728454025964',) We are sending mail with results at report@fotonower.com args[1335193049] : ('0.07702728454025964',) no score found for photo 1335193049 apple ('0.07702728454025964',) We are sending mail with results at report@fotonower.com args[1335193011] : ('0.07702728454025964',) no score found for photo 1335193011 apple ('0.07702728454025964',) We are sending mail with results at report@fotonower.com args[1335193003] : ('0.07702728454025964',) no score found for photo 1335193003 apple ('0.07702728454025964',) We are sending mail with results at report@fotonower.com args[1335192999] : ('0.07702728454025964',) no score found for photo 1335192999 apple ('0.07702728454025964',) We are sending mail with results at report@fotonower.com args[1335192939] : ('0.07702728454025964',) no score found for photo 1335192939 apple ('0.07702728454025964',) We are sending mail with results at report@fotonower.com args[1335192938] : ('0.07702728454025964',) no score found for photo 1335192938 apple ('0.07702728454025964',) We are sending mail with results at report@fotonower.com args[1335192936] : ('0.07702728454025964',) no score found for photo 1335192936 apple ('0.07702728454025964',) We are sending mail with results at report@fotonower.com refus_total : 0.07702728454025964 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20314827 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1335192933,1335193060,1335193055,1335193052,1335193049,1335193011,1335193003,1335192939,1335192938,1335192936,1335191693,1335192926,1335192136,1335191995,1335191988,1335191945,1335191696,1335192920,1335192010,1335192999) Found this number of photos: 20 begin to download photo : 1335192933 begin to download photo : 1335193011 begin to download photo : 1335191693 begin to download photo : 1335191945 download finish for photo 1335191945 begin to download photo : 1335191696 download finish for photo 1335192933 begin to download photo : 1335193060 download finish for photo 1335193011 begin to download photo : 1335193003 download finish for photo 1335191693 begin to download photo : 1335192926 download finish for photo 1335191696 begin to download photo : 1335192920 download finish for photo 1335193060 begin to download photo : 1335193055 download finish for photo 1335192926 begin to download photo : 1335192136 download finish for photo 1335192920 begin to download photo : 1335192010 download finish for photo 1335192136 begin to download photo : 1335191995 download finish for photo 1335192010 begin to download photo : 1335192999 download finish for photo 1335193055 begin to download photo : 1335193052 download finish for photo 1335191995 begin to download photo : 1335191988 download finish for photo 1335192999 download finish for photo 1335193052 begin to download photo : 1335193049 download finish for photo 1335191988 download finish for photo 1335193049 download finish for photo 1335193003 begin to download photo : 1335192939 download finish for photo 1335192939 begin to download photo : 1335192938 download finish for photo 1335192938 begin to download photo : 1335192936 download finish for photo 1335192936 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314827_07-02-2025_01_40_02.pdf results_Auto_P20314827_07-02-2025_01_40_02.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314827_07-02-2025_01_40_02.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20314827','results_Auto_P20314827_07-02-2025_01_40_02.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314827_07-02-2025_01_40_02.pdf','pdf','','0.87','0.07702728454025964') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1335193060, 1335193055, 1335193052, 1335193049, 1335193011, 1335193003, 1335192999, 1335192939, 1335192938, 1335192936] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193060', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193055', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193052', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193049', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193011', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193003', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192999', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192939', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192938', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192936', None, None, None, None, None, '2560885') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 0.015779495239257812 save_final save missing photos in datou_result : time spend for datou_step_exec : 7.461467027664185 time spend to save output : 0.016081809997558594 total time spend for step 10 : 7.477548837661743 step11:split_time_score Fri Feb 7 01:40:09 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('10', 22),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 06022025 20314827 Nombre de photos uploadées : 22 / 23040 (0%) 06022025 20314827 Nombre de photos taguées (types de déchets): 0 / 22 (0%) 06022025 20314827 Nombre de photos taguées (volume) : 0 / 22 (0%) elapsed_time : load_data_split_time_score 1.9073486328125e-06 elapsed_time : order_list_meta_photo_and_scores 5.245208740234375e-06 ?????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.001188516616821289 elapsed_time : insert_dashboard_record_day_entry 0.021656036376953125 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314812 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314813 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314814 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314815 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314816 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314817 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314818 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314819 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314820 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314821 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314822 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314823 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314824 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314825 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314826 order by id desc limit 1 Qualite : 0.07702728454025964 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314827_07-02-2025_01_40_02.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314827 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314827 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314828 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314829 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314830 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314831 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314832 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314833 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314834 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314835 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314836 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314837 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314838 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314839 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314840 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314841 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314842 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314843 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314844 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314845 order by id desc limit 1 Qualite : 0.09628085696972587 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314846_07-02-2025_01_37_46.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314846 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314846 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314847 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314848 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314849 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314850 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314851 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314852 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314853 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314854 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314855 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314856 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314857 order by id desc limit 1 NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'06022025': {'nb_upload': 22, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1335193060, 1335193055, 1335193052, 1335193049, 1335193011, 1335193003, 1335192999, 1335192939, 1335192938, 1335192936] Looping around the photos to save general results len do output : 1 /20314827Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193060', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193055', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193052', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193049', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193011', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335193003', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192999', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192939', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192938', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', '20314827', '1335192936', None, None, None, None, None, '2560885') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.015040397644042969 save_final save missing photos in datou_result : time spend for datou_step_exec : 34.32037305831909 time spend to save output : 0.015338420867919922 total time spend for step 11 : 34.33571147918701 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 183.19user 106.04system 6:20.80elapsed 75%CPU (0avgtext+0avgdata 4869688maxresident)k 2499424inputs+154720outputs (17574major+11787624minor)pagefaults 0swaps