python /home/admin/mtr/script_for_cron.py -j datou_current3 -m 10 -a ' -a 3459 -l 10 ' -s datou_current_3459 -M 0 -S 0 -U 95,95,80 import MySQLdb succeeded Import error (python version) ['/Users/moilerat/Documents/Fotonower/install/caffe/distribute/python', '/home/admin/workarea/git/Velours/python/prod', '/home/admin/workarea/install/caffe_cuda8_python3/python', '/home/admin/workarea/install/darknet', '/home/admin/workarea/git/Velours/python', '/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python', '/home/admin/mtr/.credentials', '/home/admin/workarea/install/caffe/python', '/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools', '/home/admin/workarea/git/fotonowerpip', '/home/admin/workarea/install/segment-anything', '/home/admin/workarea/git/pyfvs', '/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/admin/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages'] process id : 972799 load datou : 3459 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! Unexpected type for variable list_input_json ERROR or WARNING : can't parse json string Expecting value: line 1 column 1 (char 0) Tried to parse : chemin de la photo was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? [(photo_id, hashtag_id, hashtag_type, x0, x1, y0, y1, score, seg_temp, polygons), ...] was removed should we ? chemin de la photo was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? chemin de la photo was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? output optionnel thcl was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id, hashtag_id_0, score_0), (photo_id, hashtag_id_1, score_1), ...] was removed should we ? id de la photo (peut être local ou global) was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? chemin de la photo was removed should we ? donnée sous forme de texte was removed should we ? [ (photo_id, photo_id_loc, hashtag_type, x0, x1, y0, y1, score), ...] was removed should we ? [ (photo_id_loc, hashtag_id, hashtag_type, x0, x1, y0, y1, score, None), ...] was removed should we ? chemin de la photo was removed should we ? id de la photo (peut être local ou global) was removed should we ? chemin de la photo was removed should we ? (x0, y0, x1, y1) was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? id de la photo (peut être local ou global) was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de nombre was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? (photo_id, hashtag_id, score_max) was removed should we ? donnée sous forme de texte was removed should we ? None was removed should we ? donnée sous forme de texte was removed should we ? [ptf_id0,ptf_id1...] was removed should we ? FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) load thcls load THCL from format json or kwargs add thcl : 2456 in CacheModelConfig load THCL from format json or kwargs add thcl : 2868 in CacheModelConfig load THCL from format json or kwargs add thcl : 2896 in CacheModelConfig load pdts add pdt : 4999 in CacheModelConfig add pdt : 5288 in CacheModelConfig add pdt : 5309 in CacheModelConfig Running datou job : batch_current TODO datou_current to load to do maybe to take outside batchDatouExec updating current state to 1 we have a portfolio with more photos than limit : 22>10 please execute split_portfolio.py -i 20314827 -l 10 size over we load limit photo not treated list_input_json: [] Current got : datou_id : 3459, datou_cur_ids : ['2560885'] with mtr_portfolio_ids : ['20314827'] and first list_photo_ids : [] new path : /proc/972799/ Inside batchDatouExec : verbose : 0 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! List Step Type Loaded in datou : mask_detect, crop_condition, thcl, merge_mask_thcl_custom, rle_unique_nms_with_priority, crop_condition, ventilate_hashtags_in_portfolio, final, velours_tree, send_mail_cod, split_time_score list_input_json : [] origin We have 1 , WARNING: data may be incomplete, need to offset and complete ! BFBFBFBFBFBFBFBFBFBFwe have missing 0 photos in the step downloads : photo missing : [] try to delete the photos missing in DB length of list_filenames : 10 ; length of list_pids : 10 ; length of list_args : 10 time to download the photos : 2.041529417037964 About to test input to load we should then remove the video here, and this would fix the bug of datou_current ! Calling datou_exec Inside datou_exec : verbose : 0 number of steps : 11 step1:mask_detect Fri Feb 7 01:46:31 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Beginning of datou step mask_detect ! save_polygon : True begin detect begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5262 max_wait_temp : 1 max_wait : 0 gpu_flag : 0 2025-02-07 01:46:34.745292: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2025-02-07 01:46:34.771142: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3493065000 Hz 2025-02-07 01:46:34.773387: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f4574000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2025-02-07 01:46:34.773418: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2025-02-07 01:46:34.777319: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2025-02-07 01:46:34.941159: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xf96e360 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-02-07 01:46:34.941224: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 2080 Ti, Compute Capability 7.5 2025-02-07 01:46:34.942481: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-07 01:46:34.943221: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-07 01:46:34.946131: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-07 01:46:34.948829: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-07 01:46:34.950438: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-07 01:46:34.954098: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-07 01:46:34.955504: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-07 01:46:34.961693: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-07 01:46:34.963116: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-07 01:46:34.963189: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-07 01:46:34.963970: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-07 01:46:34.963992: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-07 01:46:34.964003: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-07 01:46:34.965266: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4804 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/admin/workarea/git/Velours/python/mtr/mask_rcnn/mask_detection.py:69: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. 2025-02-07 01:46:35.281844: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-07 01:46:35.282089: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-07 01:46:35.282121: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-07 01:46:35.282147: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-07 01:46:35.282170: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-07 01:46:35.282193: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-07 01:46:35.282213: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-07 01:46:35.282237: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-07 01:46:35.283617: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-07 01:46:35.285148: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: pciBusID: 0000:41:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2025-02-07 01:46:35.285208: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1 2025-02-07 01:46:35.285233: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-07 01:46:35.285257: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10 2025-02-07 01:46:35.285280: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10 2025-02-07 01:46:35.285303: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10 2025-02-07 01:46:35.285324: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10 2025-02-07 01:46:35.285347: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-07 01:46:35.286580: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0 2025-02-07 01:46:35.286635: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix: 2025-02-07 01:46:35.286646: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0 2025-02-07 01:46:35.286659: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N 2025-02-07 01:46:35.288015: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4804 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:41:00.0, compute capability: 7.5) Using TensorFlow backend. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:396: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version. Instructions for updating: box_ind is deprecated, use box_indices instead WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:703: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/admin/workarea/install/Mask_RCNN/model.py:729: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. Inside mask_sub_process Inside mask_detect About to load cache.load_thcl_param To do loadFromThcl(), then load ParamDescType : thcl2896 thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5309, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 16384, 25088, 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 5, 10, 19, 20, 46), datetime.datetime(2021, 5, 10, 19, 20, 46)) {'thcl': {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}, 'list_hashtags': ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'], 'list_hashtags_csv': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'svm_hashtag_type_desc': 5309, 'photo_desc_type': 5309, 'pb_hashtag_id_or_classifier': 0} list_class_names : ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] Configurations: BACKBONE resnet101 BACKBONE_SHAPES [[160 160] [ 80 80] [ 40 40] [ 20 20] [ 10 10]] BACKBONE_STRIDES [4, 8, 16, 32, 64] BATCH_SIZE 1 BBOX_STD_DEV [0.1 0.1 0.2 0.2] DETECTION_MAX_INSTANCES 100 DETECTION_MIN_CONFIDENCE 0.3 DETECTION_NMS_THRESHOLD 0.3 GPU_COUNT 1 IMAGES_PER_GPU 1 IMAGE_MAX_DIM 640 IMAGE_MIN_DIM 640 IMAGE_PADDING True IMAGE_SHAPE [640 640 3] LEARNING_MOMENTUM 0.9 LEARNING_RATE 0.001 LOSS_WEIGHTS {'rpn_class_loss': 1.0, 'rpn_bbox_loss': 1.0, 'mrcnn_class_loss': 1.0, 'mrcnn_bbox_loss': 1.0, 'mrcnn_mask_loss': 1.0} MASK_POOL_SIZE 14 MASK_SHAPE [28, 28] MAX_GT_INSTANCES 100 MEAN_PIXEL [123.7 116.8 103.9] MINI_MASK_SHAPE (56, 56) NAME learn_convoyeur_qualipapia_nantes_poly_100521_1 NUM_CLASSES 10 POOL_SIZE 7 POST_NMS_ROIS_INFERENCE 1000 POST_NMS_ROIS_TRAINING 2000 ROI_POSITIVE_RATIO 0.33 RPN_ANCHOR_RATIOS [0.5, 1, 2] RPN_ANCHOR_SCALES (16, 32, 64, 128, 256) RPN_ANCHOR_STRIDE 1 RPN_BBOX_STD_DEV [0.1 0.1 0.2 0.2] RPN_NMS_THRESHOLD 0.7 RPN_TRAIN_ANCHORS_PER_IMAGE 256 STEPS_PER_EPOCH 1000 TRAIN_ROIS_PER_IMAGE 200 USE_MINI_MASK True USE_RPN_ROIS True VALIDATION_STEPS 50 WEIGHT_DECAY 0.0001 model_param file didn't exist model_name : learn_convoyeur_qualipapia_nantes_poly_100521_1 model_type : mask_rcnn list file need : ['mask_model.h5'] file exist in s3 : ['mask_model.h5'] file manque in s3 : [] 2025-02-07 01:46:46.724142: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10 2025-02-07 01:46:46.965532: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2025-02-07 01:46:48.863384: I tensorflow/stream_executor/cuda/cuda_driver.cc:763] failed to allocate 4.00G (4294967296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory local folder : /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1 /data/models_weight/learn_convoyeur_qualipapia_nantes_poly_100521_1/mask_model.h5 size_local : 256031040 size in s3 : 256031040 create time local : 2021-08-09 05:45:48 create time in s3 : 2021-08-06 18:59:51 mask_model.h5 already exist and didn't need to update list_images length : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 26 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 22 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 29 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 14 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 10 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 15 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 19 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 18 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 18 NEW PHOTO Processing 1 images image shape: (2160, 3264, 3) min: 0.00000 max: 255.00000 molded_images shape: (1, 640, 640, 3) min: -123.70000 max: 151.10000 image_metas shape: (1, 18) min: 0.00000 max: 3264.00000 nb d'objets trouves : 14 Detection mask done ! Trying to reset tf kernel 973145 begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 381 tf kernel not reseted sub process len(results) : 10 len(list_Values) 0 None max_time_sub_proc : 3600 parent process len(results) : 10 len(list_Values) 0 process is alive finish correctly or not : True after detect begin to check gpu status inside check gpu memory l 3610 free memory gpu now : 5262 list_Values should be empty [] To do loadFromThcl(), then load ParamDescType : thcl2896 Catched exception ! Connect or reconnect ! thcls : [{'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'}] thcl {'id': 2896, 'mtr_user_id': 31, 'name': 'learn_convoyeur_qualipapia_nantes_poly_100521_1', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'background,carton_brun,carton_gris,cartonnette,kraft,autre_refus,metal,plastique,teint_dans_la_masse,environnement', 'svm_portfolios_learning': '0,0,0,0,0,0,0,0,0,0', 'photo_hashtag_type': 3663, 'photo_desc_type': 5309, 'type_classification': 'mask_rcnn', 'hashtag_id_list': '0,0,0,0,0,0,0,0,0,0'} Update svm_hashtag_type_desc : 5309 ['background', 'carton_brun', 'carton_gris', 'cartonnette', 'kraft', 'autre_refus', 'metal', 'plastique', 'teint_dans_la_masse', 'environnement'] time for calcul the mask position with numpy : 0.0013587474822998047 nb_pixel_total : 48669 time to create 1 rle with old method : 0.05404329299926758 length of segment : 295 time for calcul the mask position with numpy : 0.0027921199798583984 nb_pixel_total : 113558 time to create 1 rle with old method : 0.127455472946167 length of segment : 457 time for calcul the mask position with numpy : 0.0008318424224853516 nb_pixel_total : 35647 time to create 1 rle with old method : 0.03909730911254883 length of segment : 276 time for calcul the mask position with numpy : 0.0002739429473876953 nb_pixel_total : 4884 time to create 1 rle with old method : 0.0057337284088134766 length of segment : 144 time for calcul the mask position with numpy : 0.0008780956268310547 nb_pixel_total : 42156 time to create 1 rle with old method : 0.04644489288330078 length of segment : 238 time for calcul the mask position with numpy : 0.0007948875427246094 nb_pixel_total : 43239 time to create 1 rle with old method : 0.04812932014465332 length of segment : 332 time for calcul the mask position with numpy : 0.00023818016052246094 nb_pixel_total : 6166 time to create 1 rle with old method : 0.0073511600494384766 length of segment : 162 time for calcul the mask position with numpy : 0.0016112327575683594 nb_pixel_total : 96512 time to create 1 rle with old method : 0.10764122009277344 length of segment : 274 time for calcul the mask position with numpy : 0.004349470138549805 nb_pixel_total : 152964 time to create 1 rle with new method : 0.01320195198059082 length of segment : 504 time for calcul the mask position with numpy : 0.0019152164459228516 nb_pixel_total : 94989 time to create 1 rle with old method : 0.11031889915466309 length of segment : 324 time for calcul the mask position with numpy : 0.00013947486877441406 nb_pixel_total : 1890 time to create 1 rle with old method : 0.0024695396423339844 length of segment : 52 time for calcul the mask position with numpy : 0.00024271011352539062 nb_pixel_total : 9673 time to create 1 rle with old method : 0.010969161987304688 length of segment : 162 time for calcul the mask position with numpy : 0.0001895427703857422 nb_pixel_total : 6795 time to create 1 rle with old method : 0.008188486099243164 length of segment : 91 time for calcul the mask position with numpy : 0.00011539459228515625 nb_pixel_total : 1897 time to create 1 rle with old method : 0.0024399757385253906 length of segment : 56 time for calcul the mask position with numpy : 0.0027430057525634766 nb_pixel_total : 114020 time to create 1 rle with old method : 0.12627530097961426 length of segment : 542 time for calcul the mask position with numpy : 0.00039076805114746094 nb_pixel_total : 9887 time to create 1 rle with old method : 0.011729955673217773 length of segment : 154 time for calcul the mask position with numpy : 0.00047659873962402344 nb_pixel_total : 19230 time to create 1 rle with old method : 0.021695613861083984 length of segment : 170 time for calcul the mask position with numpy : 0.00036787986755371094 nb_pixel_total : 14358 time to create 1 rle with old method : 0.01649188995361328 length of segment : 140 time for calcul the mask position with numpy : 0.0006620883941650391 nb_pixel_total : 22689 time to create 1 rle with old method : 0.025974035263061523 length of segment : 186 time for calcul the mask position with numpy : 0.000293731689453125 nb_pixel_total : 11880 time to create 1 rle with old method : 0.013553857803344727 length of segment : 177 time for calcul the mask position with numpy : 0.0003991127014160156 nb_pixel_total : 14734 time to create 1 rle with old method : 0.017176389694213867 length of segment : 157 time for calcul the mask position with numpy : 0.00045871734619140625 nb_pixel_total : 16962 time to create 1 rle with old method : 0.019945383071899414 length of segment : 176 time for calcul the mask position with numpy : 0.0003838539123535156 nb_pixel_total : 14242 time to create 1 rle with old method : 0.016915559768676758 length of segment : 152 time for calcul the mask position with numpy : 0.00016832351684570312 nb_pixel_total : 1490 time to create 1 rle with old method : 0.001940011978149414 length of segment : 65 time for calcul the mask position with numpy : 0.0004405975341796875 nb_pixel_total : 14379 time to create 1 rle with old method : 0.016835451126098633 length of segment : 172 time for calcul the mask position with numpy : 0.0003445148468017578 nb_pixel_total : 10663 time to create 1 rle with old method : 0.012267351150512695 length of segment : 153 time for calcul the mask position with numpy : 0.00034046173095703125 nb_pixel_total : 13149 time to create 1 rle with old method : 0.015644073486328125 length of segment : 143 time for calcul the mask position with numpy : 0.0005230903625488281 nb_pixel_total : 16319 time to create 1 rle with old method : 0.01993393898010254 length of segment : 241 time for calcul the mask position with numpy : 0.0004394054412841797 nb_pixel_total : 18478 time to create 1 rle with old method : 0.022159814834594727 length of segment : 246 time for calcul the mask position with numpy : 0.0022466182708740234 nb_pixel_total : 76447 time to create 1 rle with old method : 0.08521437644958496 length of segment : 487 time for calcul the mask position with numpy : 0.0001590251922607422 nb_pixel_total : 1675 time to create 1 rle with old method : 0.002152681350708008 length of segment : 56 time for calcul the mask position with numpy : 0.0016376972198486328 nb_pixel_total : 56156 time to create 1 rle with old method : 0.06378841400146484 length of segment : 506 time for calcul the mask position with numpy : 0.0020372867584228516 nb_pixel_total : 82033 time to create 1 rle with old method : 0.09053301811218262 length of segment : 416 time for calcul the mask position with numpy : 0.0026307106018066406 nb_pixel_total : 104708 time to create 1 rle with old method : 0.11478996276855469 length of segment : 409 time for calcul the mask position with numpy : 0.0017845630645751953 nb_pixel_total : 55507 time to create 1 rle with old method : 0.06529974937438965 length of segment : 296 time for calcul the mask position with numpy : 0.0011911392211914062 nb_pixel_total : 25005 time to create 1 rle with old method : 0.029364585876464844 length of segment : 374 time for calcul the mask position with numpy : 0.0017337799072265625 nb_pixel_total : 54343 time to create 1 rle with old method : 0.07280564308166504 length of segment : 284 time for calcul the mask position with numpy : 0.0006587505340576172 nb_pixel_total : 23650 time to create 1 rle with old method : 0.026945829391479492 length of segment : 289 time for calcul the mask position with numpy : 0.00040531158447265625 nb_pixel_total : 15575 time to create 1 rle with old method : 0.023203134536743164 length of segment : 248 time for calcul the mask position with numpy : 0.0024871826171875 nb_pixel_total : 136198 time to create 1 rle with old method : 0.14978575706481934 length of segment : 332 time for calcul the mask position with numpy : 0.0003113746643066406 nb_pixel_total : 8426 time to create 1 rle with old method : 0.010329246520996094 length of segment : 82 time for calcul the mask position with numpy : 0.0001850128173828125 nb_pixel_total : 4353 time to create 1 rle with old method : 0.005419015884399414 length of segment : 57 time for calcul the mask position with numpy : 0.0010623931884765625 nb_pixel_total : 37801 time to create 1 rle with old method : 0.04332685470581055 length of segment : 322 time for calcul the mask position with numpy : 0.003949403762817383 nb_pixel_total : 125841 time to create 1 rle with old method : 0.14397525787353516 length of segment : 792 time for calcul the mask position with numpy : 0.0040705204010009766 nb_pixel_total : 179412 time to create 1 rle with new method : 0.01558232307434082 length of segment : 552 time for calcul the mask position with numpy : 0.0027015209197998047 nb_pixel_total : 113444 time to create 1 rle with old method : 0.12516117095947266 length of segment : 477 time for calcul the mask position with numpy : 0.0014276504516601562 nb_pixel_total : 45597 time to create 1 rle with old method : 0.05236673355102539 length of segment : 297 time for calcul the mask position with numpy : 0.00029850006103515625 nb_pixel_total : 9737 time to create 1 rle with old method : 0.011853933334350586 length of segment : 148 time for calcul the mask position with numpy : 0.0001690387725830078 nb_pixel_total : 2494 time to create 1 rle with old method : 0.0031576156616210938 length of segment : 75 time for calcul the mask position with numpy : 0.0016360282897949219 nb_pixel_total : 70139 time to create 1 rle with old method : 0.09398961067199707 length of segment : 489 time for calcul the mask position with numpy : 0.0026547908782958984 nb_pixel_total : 65877 time to create 1 rle with old method : 0.07229733467102051 length of segment : 360 time for calcul the mask position with numpy : 0.003397226333618164 nb_pixel_total : 80725 time to create 1 rle with old method : 0.09061193466186523 length of segment : 383 time for calcul the mask position with numpy : 0.0024552345275878906 nb_pixel_total : 39044 time to create 1 rle with old method : 0.04379105567932129 length of segment : 290 time for calcul the mask position with numpy : 0.002821683883666992 nb_pixel_total : 58490 time to create 1 rle with old method : 0.0720052719116211 length of segment : 370 time for calcul the mask position with numpy : 0.004280567169189453 nb_pixel_total : 102737 time to create 1 rle with old method : 0.1277298927307129 length of segment : 338 time for calcul the mask position with numpy : 0.0006742477416992188 nb_pixel_total : 14011 time to create 1 rle with old method : 0.01580524444580078 length of segment : 162 time for calcul the mask position with numpy : 0.0006997585296630859 nb_pixel_total : 9392 time to create 1 rle with old method : 0.011398792266845703 length of segment : 225 time for calcul the mask position with numpy : 0.00045752525329589844 nb_pixel_total : 16542 time to create 1 rle with old method : 0.018903493881225586 length of segment : 157 time for calcul the mask position with numpy : 0.0009367465972900391 nb_pixel_total : 19989 time to create 1 rle with old method : 0.023230552673339844 length of segment : 212 time for calcul the mask position with numpy : 0.0009746551513671875 nb_pixel_total : 30015 time to create 1 rle with old method : 0.037377119064331055 length of segment : 185 time for calcul the mask position with numpy : 0.0002636909484863281 nb_pixel_total : 2105 time to create 1 rle with old method : 0.003248453140258789 length of segment : 33 time for calcul the mask position with numpy : 0.00021338462829589844 nb_pixel_total : 3505 time to create 1 rle with old method : 0.004239559173583984 length of segment : 62 time for calcul the mask position with numpy : 0.0010709762573242188 nb_pixel_total : 72794 time to create 1 rle with old method : 0.08156442642211914 length of segment : 328 time for calcul the mask position with numpy : 0.003569364547729492 nb_pixel_total : 85585 time to create 1 rle with old method : 0.09560036659240723 length of segment : 364 time for calcul the mask position with numpy : 0.0027625560760498047 nb_pixel_total : 80386 time to create 1 rle with old method : 0.09092521667480469 length of segment : 353 time for calcul the mask position with numpy : 0.0029435157775878906 nb_pixel_total : 66382 time to create 1 rle with old method : 0.0734095573425293 length of segment : 366 time for calcul the mask position with numpy : 0.001184701919555664 nb_pixel_total : 27977 time to create 1 rle with old method : 0.031397104263305664 length of segment : 253 time for calcul the mask position with numpy : 0.0019240379333496094 nb_pixel_total : 69590 time to create 1 rle with old method : 0.07842254638671875 length of segment : 304 time for calcul the mask position with numpy : 0.0025818347930908203 nb_pixel_total : 63362 time to create 1 rle with old method : 0.09204506874084473 length of segment : 372 time for calcul the mask position with numpy : 0.0015323162078857422 nb_pixel_total : 62042 time to create 1 rle with old method : 0.06943106651306152 length of segment : 355 time for calcul the mask position with numpy : 0.002123594284057617 nb_pixel_total : 44361 time to create 1 rle with old method : 0.058144330978393555 length of segment : 442 time for calcul the mask position with numpy : 0.00439000129699707 nb_pixel_total : 130103 time to create 1 rle with old method : 0.15558147430419922 length of segment : 595 time for calcul the mask position with numpy : 0.0018711090087890625 nb_pixel_total : 36451 time to create 1 rle with old method : 0.04195046424865723 length of segment : 176 time for calcul the mask position with numpy : 0.0005121231079101562 nb_pixel_total : 6691 time to create 1 rle with old method : 0.01271200180053711 length of segment : 124 time for calcul the mask position with numpy : 0.009679317474365234 nb_pixel_total : 247542 time to create 1 rle with new method : 0.018463134765625 length of segment : 776 time for calcul the mask position with numpy : 0.01058197021484375 nb_pixel_total : 256810 time to create 1 rle with new method : 0.05383944511413574 length of segment : 905 time for calcul the mask position with numpy : 0.002772092819213867 nb_pixel_total : 76119 time to create 1 rle with old method : 0.08601164817810059 length of segment : 508 time for calcul the mask position with numpy : 0.0012166500091552734 nb_pixel_total : 42875 time to create 1 rle with old method : 0.049367666244506836 length of segment : 400 time for calcul the mask position with numpy : 0.005554676055908203 nb_pixel_total : 327884 time to create 1 rle with new method : 0.011983394622802734 length of segment : 504 time for calcul the mask position with numpy : 0.001680135726928711 nb_pixel_total : 29067 time to create 1 rle with old method : 0.03368878364562988 length of segment : 292 time for calcul the mask position with numpy : 0.0001857280731201172 nb_pixel_total : 958 time to create 1 rle with old method : 0.0012650489807128906 length of segment : 51 time for calcul the mask position with numpy : 0.0013546943664550781 nb_pixel_total : 36518 time to create 1 rle with old method : 0.04235720634460449 length of segment : 285 time for calcul the mask position with numpy : 0.0011341571807861328 nb_pixel_total : 62610 time to create 1 rle with old method : 0.07021331787109375 length of segment : 296 time for calcul the mask position with numpy : 0.010826349258422852 nb_pixel_total : 417538 time to create 1 rle with new method : 0.04265999794006348 length of segment : 485 time for calcul the mask position with numpy : 0.007659435272216797 nb_pixel_total : 443384 time to create 1 rle with new method : 0.03030872344970703 length of segment : 517 time for calcul the mask position with numpy : 0.0044708251953125 nb_pixel_total : 176995 time to create 1 rle with new method : 0.010836601257324219 length of segment : 467 time for calcul the mask position with numpy : 0.2678661346435547 nb_pixel_total : 3269366 time to create 1 rle with new method : 0.3811802864074707 length of segment : 3514 time for calcul the mask position with numpy : 0.00022172927856445312 nb_pixel_total : 6687 time to create 1 rle with old method : 0.010489463806152344 length of segment : 122 time for calcul the mask position with numpy : 0.00501251220703125 nb_pixel_total : 242807 time to create 1 rle with new method : 0.012977838516235352 length of segment : 615 time for calcul the mask position with numpy : 0.0004830360412597656 nb_pixel_total : 17494 time to create 1 rle with old method : 0.020464658737182617 length of segment : 198 time for calcul the mask position with numpy : 0.0009086132049560547 nb_pixel_total : 44532 time to create 1 rle with old method : 0.05051565170288086 length of segment : 476 time for calcul the mask position with numpy : 0.0003554821014404297 nb_pixel_total : 12739 time to create 1 rle with old method : 0.014470338821411133 length of segment : 143 time for calcul the mask position with numpy : 0.0006275177001953125 nb_pixel_total : 36938 time to create 1 rle with old method : 0.04110455513000488 length of segment : 375 time for calcul the mask position with numpy : 0.00032901763916015625 nb_pixel_total : 16506 time to create 1 rle with old method : 0.019527912139892578 length of segment : 298 time for calcul the mask position with numpy : 0.0020842552185058594 nb_pixel_total : 87178 time to create 1 rle with old method : 0.1002206802368164 length of segment : 375 time for calcul the mask position with numpy : 0.0010385513305664062 nb_pixel_total : 37560 time to create 1 rle with old method : 0.04307222366333008 length of segment : 307 time for calcul the mask position with numpy : 0.0013720989227294922 nb_pixel_total : 51733 time to create 1 rle with old method : 0.064666748046875 length of segment : 323 time for calcul the mask position with numpy : 0.000270843505859375 nb_pixel_total : 8488 time to create 1 rle with old method : 0.009877681732177734 length of segment : 123 time for calcul the mask position with numpy : 0.002470254898071289 nb_pixel_total : 80240 time to create 1 rle with old method : 0.0935971736907959 length of segment : 305 time for calcul the mask position with numpy : 0.00029349327087402344 nb_pixel_total : 3299 time to create 1 rle with old method : 0.00490570068359375 length of segment : 94 time for calcul the mask position with numpy : 0.004488945007324219 nb_pixel_total : 164581 time to create 1 rle with new method : 0.006248950958251953 length of segment : 582 time for calcul the mask position with numpy : 0.002862691879272461 nb_pixel_total : 60252 time to create 1 rle with old method : 0.07399129867553711 length of segment : 435 time for calcul the mask position with numpy : 0.0034666061401367188 nb_pixel_total : 111731 time to create 1 rle with old method : 0.12396430969238281 length of segment : 297 time for calcul the mask position with numpy : 0.0003104209899902344 nb_pixel_total : 4806 time to create 1 rle with old method : 0.005768775939941406 length of segment : 83 time for calcul the mask position with numpy : 0.0010924339294433594 nb_pixel_total : 49662 time to create 1 rle with old method : 0.057791948318481445 length of segment : 198 time for calcul the mask position with numpy : 0.0001895427703857422 nb_pixel_total : 2233 time to create 1 rle with old method : 0.0028269290924072266 length of segment : 112 time for calcul the mask position with numpy : 0.002086162567138672 nb_pixel_total : 57697 time to create 1 rle with old method : 0.06606125831604004 length of segment : 343 time for calcul the mask position with numpy : 0.00022864341735839844 nb_pixel_total : 2526 time to create 1 rle with old method : 0.0033266544342041016 length of segment : 48 time for calcul the mask position with numpy : 0.0007164478302001953 nb_pixel_total : 19138 time to create 1 rle with old method : 0.022504091262817383 length of segment : 258 time for calcul the mask position with numpy : 0.0014758110046386719 nb_pixel_total : 51044 time to create 1 rle with old method : 0.09313106536865234 length of segment : 200 time for calcul the mask position with numpy : 0.0018579959869384766 nb_pixel_total : 29859 time to create 1 rle with old method : 0.055570125579833984 length of segment : 226 time for calcul the mask position with numpy : 0.0020477771759033203 nb_pixel_total : 31312 time to create 1 rle with old method : 0.06006813049316406 length of segment : 342 time for calcul the mask position with numpy : 0.001444101333618164 nb_pixel_total : 35206 time to create 1 rle with old method : 0.05729103088378906 length of segment : 222 time for calcul the mask position with numpy : 0.0010876655578613281 nb_pixel_total : 35918 time to create 1 rle with old method : 0.04094886779785156 length of segment : 214 time for calcul the mask position with numpy : 0.00021576881408691406 nb_pixel_total : 3803 time to create 1 rle with old method : 0.004637718200683594 length of segment : 75 time for calcul the mask position with numpy : 0.0016398429870605469 nb_pixel_total : 63888 time to create 1 rle with old method : 0.07114028930664062 length of segment : 310 time for calcul the mask position with numpy : 0.00026702880859375 nb_pixel_total : 3121 time to create 1 rle with old method : 0.003760814666748047 length of segment : 120 time for calcul the mask position with numpy : 0.001878976821899414 nb_pixel_total : 48293 time to create 1 rle with old method : 0.0544586181640625 length of segment : 292 time for calcul the mask position with numpy : 0.0002276897430419922 nb_pixel_total : 3450 time to create 1 rle with old method : 0.004332542419433594 length of segment : 60 time for calcul the mask position with numpy : 0.00020623207092285156 nb_pixel_total : 4086 time to create 1 rle with old method : 0.004751443862915039 length of segment : 105 time for calcul the mask position with numpy : 0.00029969215393066406 nb_pixel_total : 3872 time to create 1 rle with old method : 0.004674434661865234 length of segment : 100 time for calcul the mask position with numpy : 0.0009214878082275391 nb_pixel_total : 44712 time to create 1 rle with old method : 0.0541231632232666 length of segment : 273 time for calcul the mask position with numpy : 0.0010616779327392578 nb_pixel_total : 39329 time to create 1 rle with old method : 0.044594526290893555 length of segment : 248 time for calcul the mask position with numpy : 0.00021505355834960938 nb_pixel_total : 1712 time to create 1 rle with old method : 0.002127408981323242 length of segment : 61 time for calcul the mask position with numpy : 0.005526304244995117 nb_pixel_total : 179372 time to create 1 rle with new method : 0.008983850479125977 length of segment : 537 time for calcul the mask position with numpy : 0.0037605762481689453 nb_pixel_total : 197942 time to create 1 rle with new method : 0.008847951889038086 length of segment : 546 time for calcul the mask position with numpy : 0.0009329319000244141 nb_pixel_total : 22450 time to create 1 rle with old method : 0.026035785675048828 length of segment : 197 time for calcul the mask position with numpy : 0.0010249614715576172 nb_pixel_total : 26950 time to create 1 rle with old method : 0.03216743469238281 length of segment : 270 time for calcul the mask position with numpy : 0.0010974407196044922 nb_pixel_total : 34444 time to create 1 rle with old method : 0.04154467582702637 length of segment : 264 time for calcul the mask position with numpy : 0.0007345676422119141 nb_pixel_total : 12541 time to create 1 rle with old method : 0.016304731369018555 length of segment : 208 time for calcul the mask position with numpy : 0.0003361701965332031 nb_pixel_total : 6940 time to create 1 rle with old method : 0.008508920669555664 length of segment : 80 time for calcul the mask position with numpy : 0.0006353855133056641 nb_pixel_total : 9543 time to create 1 rle with old method : 0.011966228485107422 length of segment : 107 time for calcul the mask position with numpy : 0.0007658004760742188 nb_pixel_total : 16155 time to create 1 rle with old method : 0.02047133445739746 length of segment : 198 time for calcul the mask position with numpy : 0.0003504753112792969 nb_pixel_total : 2603 time to create 1 rle with old method : 0.0031881332397460938 length of segment : 101 time for calcul the mask position with numpy : 0.00023627281188964844 nb_pixel_total : 3125 time to create 1 rle with old method : 0.003841876983642578 length of segment : 89 time for calcul the mask position with numpy : 0.0011882781982421875 nb_pixel_total : 37558 time to create 1 rle with old method : 0.05197954177856445 length of segment : 328 time for calcul the mask position with numpy : 0.0003743171691894531 nb_pixel_total : 5974 time to create 1 rle with old method : 0.009103775024414062 length of segment : 103 time for calcul the mask position with numpy : 0.0001678466796875 nb_pixel_total : 1550 time to create 1 rle with old method : 0.00203704833984375 length of segment : 51 time for calcul the mask position with numpy : 0.004303455352783203 nb_pixel_total : 156536 time to create 1 rle with new method : 0.0066165924072265625 length of segment : 395 time for calcul the mask position with numpy : 0.0010166168212890625 nb_pixel_total : 21621 time to create 1 rle with old method : 0.03400158882141113 length of segment : 127 time for calcul the mask position with numpy : 0.0002827644348144531 nb_pixel_total : 2202 time to create 1 rle with old method : 0.003039121627807617 length of segment : 52 time for calcul the mask position with numpy : 0.0011866092681884766 nb_pixel_total : 46349 time to create 1 rle with old method : 0.051618337631225586 length of segment : 277 time for calcul the mask position with numpy : 0.0008983612060546875 nb_pixel_total : 17605 time to create 1 rle with old method : 0.020703554153442383 length of segment : 314 time for calcul the mask position with numpy : 0.00019884109497070312 nb_pixel_total : 2103 time to create 1 rle with old method : 0.0025975704193115234 length of segment : 57 time for calcul the mask position with numpy : 0.0011050701141357422 nb_pixel_total : 34969 time to create 1 rle with old method : 0.04075312614440918 length of segment : 278 time for calcul the mask position with numpy : 0.00036644935607910156 nb_pixel_total : 7895 time to create 1 rle with old method : 0.009301900863647461 length of segment : 97 time for calcul the mask position with numpy : 0.0013430118560791016 nb_pixel_total : 55716 time to create 1 rle with old method : 0.06619691848754883 length of segment : 284 time for calcul the mask position with numpy : 0.11313104629516602 nb_pixel_total : 1893985 time to create 1 rle with new method : 0.244492769241333 length of segment : 2649 time for calcul the mask position with numpy : 0.0021314620971679688 nb_pixel_total : 42511 time to create 1 rle with old method : 0.049251556396484375 length of segment : 312 time for calcul the mask position with numpy : 0.0007748603820800781 nb_pixel_total : 33587 time to create 1 rle with old method : 0.03927946090698242 length of segment : 200 time for calcul the mask position with numpy : 0.0010743141174316406 nb_pixel_total : 20422 time to create 1 rle with old method : 0.023380041122436523 length of segment : 152 time for calcul the mask position with numpy : 0.004024028778076172 nb_pixel_total : 116335 time to create 1 rle with old method : 0.1274557113647461 length of segment : 470 time for calcul the mask position with numpy : 0.003392457962036133 nb_pixel_total : 101624 time to create 1 rle with old method : 0.11198711395263672 length of segment : 332 time for calcul the mask position with numpy : 0.00034546852111816406 nb_pixel_total : 2800 time to create 1 rle with old method : 0.0035467147827148438 length of segment : 86 time for calcul the mask position with numpy : 0.0029315948486328125 nb_pixel_total : 72491 time to create 1 rle with old method : 0.08076739311218262 length of segment : 322 time for calcul the mask position with numpy : 0.0027480125427246094 nb_pixel_total : 68049 time to create 1 rle with old method : 0.07623004913330078 length of segment : 404 time for calcul the mask position with numpy : 0.007123231887817383 nb_pixel_total : 287009 time to create 1 rle with new method : 0.01196599006652832 length of segment : 529 time for calcul the mask position with numpy : 0.00091552734375 nb_pixel_total : 23202 time to create 1 rle with old method : 0.026015281677246094 length of segment : 177 time for calcul the mask position with numpy : 0.0041730403900146484 nb_pixel_total : 118566 time to create 1 rle with old method : 0.13587379455566406 length of segment : 582 time for calcul the mask position with numpy : 0.0001690387725830078 nb_pixel_total : 6923 time to create 1 rle with old method : 0.007743120193481445 length of segment : 85 time for calcul the mask position with numpy : 0.0002281665802001953 nb_pixel_total : 2419 time to create 1 rle with old method : 0.002893686294555664 length of segment : 94 time for calcul the mask position with numpy : 0.0003516674041748047 nb_pixel_total : 9672 time to create 1 rle with old method : 0.011210918426513672 length of segment : 100 time for calcul the mask position with numpy : 0.0010595321655273438 nb_pixel_total : 46842 time to create 1 rle with old method : 0.05110430717468262 length of segment : 258 time for calcul the mask position with numpy : 0.0006434917449951172 nb_pixel_total : 24198 time to create 1 rle with old method : 0.027388572692871094 length of segment : 184 time for calcul the mask position with numpy : 0.004694938659667969 nb_pixel_total : 128717 time to create 1 rle with old method : 0.15033745765686035 length of segment : 936 time for calcul the mask position with numpy : 0.0003292560577392578 nb_pixel_total : 4305 time to create 1 rle with old method : 0.0051670074462890625 length of segment : 90 time for calcul the mask position with numpy : 0.0050373077392578125 nb_pixel_total : 117092 time to create 1 rle with old method : 0.13363909721374512 length of segment : 795 time for calcul the mask position with numpy : 0.0047261714935302734 nb_pixel_total : 33283 time to create 1 rle with old method : 0.0375666618347168 length of segment : 426 time for calcul the mask position with numpy : 0.0012536048889160156 nb_pixel_total : 34470 time to create 1 rle with old method : 0.03903913497924805 length of segment : 611 time for calcul the mask position with numpy : 0.0005500316619873047 nb_pixel_total : 425 time to create 1 rle with old method : 0.0007987022399902344 length of segment : 49 time for calcul the mask position with numpy : 0.00041484832763671875 nb_pixel_total : 11498 time to create 1 rle with old method : 0.013384103775024414 length of segment : 227 time spent for convertir_results : 18.023189544677734 Inside saveOutput : final : False verbose : 0 eke 12-6-18 : saveMask need to be cleaned for new output ! Number saved : None batch 1 Loaded 171 chid ids of type : 3663 Number RLEs to save : 53026 save missing photos in datou_result : time spend for datou_step_exec : 67.53270888328552 time spend to save output : 7.8020548820495605 total time spend for step 1 : 75.33476376533508 step2:crop_condition Fri Feb 7 01:47:46 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure Loading chi in step crop with photo_hashtag_type : 3663 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 171 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 16 About to insert : list_path_to_insert length 16 new photo from crops ! we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 5 About to insert : list_path_to_insert length 5 new photo from crops ! we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 59 About to insert : list_path_to_insert length 59 new photo from crops ! we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 7 About to insert : list_path_to_insert length 7 new photo from crops ! we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! we have both polygon and rles Next one ! map_result returned by crop_photo_return_map_crop : length : 11 About to insert : list_path_to_insert length 11 new photo from crops ! we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1335192933, 1335192926, 1335192920, 1335192136, 1335192010, 1335192000, 1335191995, 1335191988, 1335191949, 1335191945] Looping around the photos to save general results len do output : 107 /-3658511157Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511152Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511166Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511179Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511198Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511233Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511253Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511269Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511304Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511149Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511187Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511204Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511203Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511197Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511213Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511226Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511225Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511230Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511250Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511247Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511256Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511265Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511282Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511299Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511305Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511163Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511207Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511208Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511195Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511242Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511143Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511148Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511150Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511141Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511145Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511161Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511159Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511183Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511170Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511177Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511180Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511185Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511174Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511186Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511178Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511175Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511184Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511172Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511168Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511173Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511211Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511200Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511205Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511196Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511192Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511193Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511212Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511199Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511191Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511190Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511194Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511206Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511217Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511224Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511223Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511235Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511229Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511234Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511249Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511241Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511251Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511258Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511268Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511254Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511262Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511266Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511267Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511263Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511280Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511279Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511289Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511290Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511293Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511285Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511296Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511297Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511287Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511309Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511308Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511156Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511228Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511232Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511244Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511276Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511277Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511291Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511146Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511155Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511142Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511216Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511220Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511222Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511231Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511245Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511243Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511292Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /-3658511307Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192933', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192926', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192920', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192136', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192010', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192000', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191995', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191988', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191949', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191945', None, None, None, None, None, '2560885') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 331 time used for this insertion : 0.03764152526855469 save_final save missing photos in datou_result : time spend for datou_step_exec : 44.52565336227417 time spend to save output : 0.04573464393615723 total time spend for step 2 : 44.57138800621033 step3:thcl Fri Feb 7 01:48:31 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step Thcl ! nombre de thcls : 2 we are using the classfication for multi_thcl [2456, 2868] time to import caffe and check if the image exist : 0.030173063278198242 time to convert the images to numpy array : 0.40269994735717773 time to import caffe and check if the image exist : 0.021477222442626953 time to convert the images to numpy array : 0.4324064254760742 time to import caffe and check if the image exist : 0.02581000328063965 time to convert the images to numpy array : 0.5630383491516113 time to import caffe and check if the image exist : 0.023297786712646484 time to convert the images to numpy array : 0.5738945007324219 time to import caffe and check if the image exist : 0.02667689323425293 time to convert the images to numpy array : 0.5729706287384033 time to import caffe and check if the image exist : 0.027092933654785156 time to convert the images to numpy array : 0.6151382923126221 time to import caffe and check if the image exist : 0.024707794189453125 time to convert the images to numpy array : 0.6253304481506348 time to import caffe and check if the image exist : 0.023358583450317383 time to convert the images to numpy array : 0.635514497756958 time to import caffe and check if the image exist : 0.01641392707824707 time to convert the images to numpy array : 0.655733585357666 time to import caffe and check if the image exist : 0.024744033813476562 time to convert the images to numpy array : 0.6626999378204346 total time to convert the images to numpy array : 2.2393293380737305 list photo_ids error: [] list photo_ids correct : [-3658511157, -3658511152, -3658511166, -3658511179, -3658511198, -3658511233, -3658511253, -3658511269, -3658511304, -3658511149, -3658511187, -3658511308, -3658511156, -3658511228, -3658511232, -3658511244, -3658511276, -3658511277, -3658511291, -3658511146, -3658511155, -3658511142, -3658511178, -3658511175, -3658511184, -3658511172, -3658511168, -3658511173, -3658511211, -3658511200, -3658511205, -3658511196, -3658511192, -3658511263, -3658511280, -3658511279, -3658511289, -3658511290, -3658511293, -3658511285, -3658511296, -3658511297, -3658511287, -3658511309, -3658511141, -3658511145, -3658511161, -3658511159, -3658511183, -3658511170, -3658511177, -3658511180, -3658511185, -3658511174, -3658511186, -3658511229, -3658511234, -3658511249, -3658511241, -3658511251, -3658511258, -3658511268, -3658511254, -3658511262, -3658511266, -3658511267, -3658511282, -3658511299, -3658511305, -3658511163, -3658511207, -3658511208, -3658511195, -3658511242, -3658511143, -3658511148, -3658511150, -3658511204, -3658511203, -3658511197, -3658511213, -3658511226, -3658511225, -3658511230, -3658511250, -3658511247, -3658511256, -3658511265, -3658511216, -3658511220, -3658511222, -3658511231, -3658511245, -3658511243, -3658511292, -3658511307, -3658511193, -3658511212, -3658511199, -3658511191, -3658511190, -3658511194, -3658511206, -3658511217, -3658511224, -3658511223, -3658511235] number of photos to traite : 107 try to delete the photos incorrect in DB tagging for thcl : 2456 To do loadFromThcl(), then load ParamDescType : thcl2456 thcls : [{'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'}] thcl {'id': 2456, 'mtr_user_id': 31, 'name': 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'papier,refus', 'svm_portfolios_learning': '3028087,3028251', 'photo_hashtag_type': 3049, 'photo_desc_type': 4999, 'type_classification': 'caffe', 'hashtag_id_list': '492668766,538914404'} Update svm_hashtag_type_desc : 4999 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) To loadFromThcl() : net_4999 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 5262 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (4999, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 16384, 25088, 'learn_qualipapia_papier_refus_from_vlg_data_aug', 'res5b', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2020, 10, 23, 14, 27, 22), datetime.datetime(2020, 10, 23, 14, 27, 22)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_qualipapia_papier_refus_from_vlg_data_aug Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_qualipapia_papier_refus_from_vlg_data_aug model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel size_local : 44972172 size in s3 : 44972172 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 caffemodel already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt size_local : 17311 size in s3 : 17311 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:51 mean.npy already exist and didn't need to update /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/synset_words.txt size_local : 57 size in s3 : 57 create time local : 2021-08-09 05:55:48 create time in s3 : 2021-08-06 19:28:49 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/deploy.prototxt caffemodel_filename : /data/models_weight/learn_qualipapia_papier_refus_from_vlg_data_aug/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 4808 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.3976821899414062 time used to do the prediction : 0.3709726333618164 we don't save the descriptors for this thcl 2456 tagging for thcl : 2868 To do loadFromThcl(), then load ParamDescType : thcl2868 thcls : [{'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2868, 'mtr_user_id': 31, 'name': 'learn_papier_nantes_300421', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3752117,3752118,3752123,3752106,3752116,3752124,3752119,3581575,3486029,3752122', 'photo_hashtag_type': 3632, 'photo_desc_type': 5288, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5288 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) To loadFromThcl() : net_5288 begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 4242 max_wait_temp : 1 max_wait : 0 FOUND : 1 Here is data_from_sql_as_vec to set the ParamDescriptorType : (5288, 'learn_papier_nantes_300421', 512, 512, 'learn_papier_nantes_300421', 'pool5', 10.0, None, None, 256, None, 0, None, 8, None, None, -1000.0, 1, datetime.datetime(2021, 4, 30, 17, 9, 41), datetime.datetime(2021, 4, 30, 17, 9, 41)) None mean_file_type : mean_file_path : prototxt_file_path : model : learn_papier_nantes_300421 Inside get_net Inside get_net before cache_data_model model_param file didn't exist Inside get_net before CDM.load_model_par_type model_name : learn_papier_nantes_300421 model_type : caffe list file need : ['caffemodel', 'deploy_conv_normal.prototxt', 'deploy_fc.prototxt', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file exist in s3 : ['caffemodel', 'deploy.prototxt', 'mean.npy', 'synset_words.txt'] file manque in s3 : ['deploy_conv_normal.prototxt', 'deploy_fc.prototxt'] local folder : /data/models_weight/learn_papier_nantes_300421 /data/models_weight/learn_papier_nantes_300421/caffemodel size_local : 44791983 size in s3 : 44791983 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:13 caffemodel already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/deploy.prototxt size_local : 17255 size in s3 : 17255 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:12 deploy.prototxt already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/mean.npy size_local : 1572992 size in s3 : 1572992 create time local : 2021-08-09 05:55:59 create time in s3 : 2021-08-06 19:22:14 mean.npy already exist and didn't need to update /data/models_weight/learn_papier_nantes_300421/synset_words.txt size_local : 331 size in s3 : 331 create time local : 2021-08-09 05:56:00 create time in s3 : 2021-08-06 19:22:12 synset_words.txt already exist and didn't need to update Inside get_net after CDM.load_model_par_type After if not only_with_local_cache: /home/admin/workarea/install/caffe_cuda8_python3/python/:/home/admin/workarea/install/darknet/:/home/admin/workarea/git/Velours/python:/home/admin/workarea/install/caffe_frcnn_python3/py-faster-rcnn/caffe-fast-rcnn/python:/home/admin/mtr/.credentials:/home/admin/workarea/install/caffe/python:/home/admin/workarea/install/caffe_frcnn/py-faster-rcnn/tools/:/home/admin/workarea/git/fotonowerpip/:/home/admin/workarea/install/segment-anything:/home/admin//workarea/git/pyfvs/ Here before set mode gpu Doing nothing but we could set mode gpu after set mode gpu prototxt_filename : /data/models_weight/learn_papier_nantes_300421/deploy.prototxt caffemodel_filename : /data/models_weight/learn_papier_nantes_300421/caffemodel now we set caffe to gpu mode before predict begin to check gpu status inside check gpu memory l 3637 free memory gpu now : 4201 max_wait_temp : 1 max_wait : 0 dict_keys(['prob']) time used to do the prepocess of the images : 1.3541152477264404 time used to do the prediction : 0.22028541564941406 we don't save the descriptors for this thcl 2868 Inside saveOutput : final : False verbose : 0 time used to find the portfolios of the photos test new format of the output of the step_thcl begin to insert list_values into class_photo_scores : length of list_valuse in save_photo_hashtag_id_thcl_score : 0 time used for this insertion : 2.4557113647460938e-05 save missing photos in datou_result : time spend for datou_step_exec : 14.229184627532959 time spend to save output : 0.0010993480682373047 total time spend for step 3 : 14.230283975601196 step4:merge_mask_thcl_custom Fri Feb 7 01:48:45 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 1 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step merge_mask_thcl_custom batch 1 Loaded 171 chid ids of type : 3663 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++As expected we have just one thcl present End of step merge_mask_thcl_custom Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : merge_mask_thcl_custom we use saveGeneral [1335192933, 1335192926, 1335192920, 1335192136, 1335192010, 1335192000, 1335191995, 1335191988, 1335191949, 1335191945] Looping around the photos to save general results len do output : 10 /1335192933Didn't retrieve data .Didn't retrieve data . /1335192926Didn't retrieve data .Didn't retrieve data . /1335192920Didn't retrieve data .Didn't retrieve data . /1335192136Didn't retrieve data .Didn't retrieve data . /1335192010Didn't retrieve data .Didn't retrieve data . /1335192000Didn't retrieve data .Didn't retrieve data . /1335191995Didn't retrieve data .Didn't retrieve data . /1335191988Didn't retrieve data .Didn't retrieve data . /1335191949Didn't retrieve data .Didn't retrieve data . /1335191945Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192933', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192926', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192920', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192136', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192010', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192000', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191995', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191988', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191949', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191945', None, None, None, None, None, '2560885') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.21526169776916504 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.21396088600158691 time spend to save output : 0.21585536003112793 total time spend for step 4 : 0.42981624603271484 step5:rle_unique_nms_with_priority Fri Feb 7 01:48:46 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure Begin step rle-unique-nms nb_obj : 23 nb_hashtags : 7 time to prepare the origin masks : 13.365498781204224 time for calcul the mask position with numpy : 0.4976935386657715 nb_pixel_total : 6192139 time to create 1 rle with new method : 2.532794713973999 time for calcul the mask position with numpy : 0.03314805030822754 nb_pixel_total : 152907 time to create 1 rle with new method : 0.45024609565734863 time for calcul the mask position with numpy : 0.03642988204956055 nb_pixel_total : 43145 time to create 1 rle with old method : 0.05306649208068848 time for calcul the mask position with numpy : 0.03866720199584961 nb_pixel_total : 35570 time to create 1 rle with old method : 0.03691291809082031 time for calcul the mask position with numpy : 0.03792309761047363 nb_pixel_total : 14231 time to create 1 rle with old method : 0.015489578247070312 time for calcul the mask position with numpy : 0.03729104995727539 nb_pixel_total : 9866 time to create 1 rle with old method : 0.011149406433105469 time for calcul the mask position with numpy : 0.03796267509460449 nb_pixel_total : 48422 time to create 1 rle with old method : 0.05357003211975098 time for calcul the mask position with numpy : 0.039916276931762695 nb_pixel_total : 113110 time to create 1 rle with old method : 0.1226954460144043 time for calcul the mask position with numpy : 0.03729367256164551 nb_pixel_total : 315 time to create 1 rle with old method : 0.0004780292510986328 time for calcul the mask position with numpy : 0.04151296615600586 nb_pixel_total : 94499 time to create 1 rle with old method : 0.10321283340454102 time for calcul the mask position with numpy : 0.036435842514038086 nb_pixel_total : 11820 time to create 1 rle with old method : 0.012906789779663086 time for calcul the mask position with numpy : 0.03507065773010254 nb_pixel_total : 14280 time to create 1 rle with old method : 0.01583075523376465 time for calcul the mask position with numpy : 0.0389552116394043 nb_pixel_total : 14646 time to create 1 rle with old method : 0.014888525009155273 time for calcul the mask position with numpy : 0.03830862045288086 nb_pixel_total : 1874 time to create 1 rle with old method : 0.0021576881408691406 time for calcul the mask position with numpy : 0.03372025489807129 nb_pixel_total : 42123 time to create 1 rle with old method : 0.04550457000732422 time for calcul the mask position with numpy : 0.03661918640136719 nb_pixel_total : 6783 time to create 1 rle with old method : 0.007702350616455078 time for calcul the mask position with numpy : 0.038916826248168945 nb_pixel_total : 1881 time to create 1 rle with old method : 0.0021712779998779297 time for calcul the mask position with numpy : 0.038931846618652344 nb_pixel_total : 3483 time to create 1 rle with old method : 0.004195213317871094 time for calcul the mask position with numpy : 0.04015755653381348 nb_pixel_total : 6151 time to create 1 rle with old method : 0.007234334945678711 time for calcul the mask position with numpy : 0.03973126411437988 nb_pixel_total : 113260 time to create 1 rle with old method : 0.1206514835357666 time for calcul the mask position with numpy : 0.038703203201293945 nb_pixel_total : 9659 time to create 1 rle with old method : 0.010933399200439453 time for calcul the mask position with numpy : 0.04515409469604492 nb_pixel_total : 4850 time to create 1 rle with old method : 0.0055048465728759766 time for calcul the mask position with numpy : 0.04126930236816406 nb_pixel_total : 96065 time to create 1 rle with old method : 0.10260629653930664 time for calcul the mask position with numpy : 0.039678096771240234 nb_pixel_total : 19161 time to create 1 rle with old method : 0.02033686637878418 create new chi : 5.207198619842529 time to delete rle : 0.014816999435424805 batch 1 Loaded 24 chid ids of type : 3726 Number RLEs to save : 12230 TO DO : save crop sub photo not yet done ! save time : 2.6279194355010986 nb_obj : 22 nb_hashtags : 7 time to prepare the origin masks : 13.995715618133545 time for calcul the mask position with numpy : 0.31483960151672363 nb_pixel_total : 5905409 time to create 1 rle with new method : 0.6505684852600098 time for calcul the mask position with numpy : 0.025619029998779297 nb_pixel_total : 10057 time to create 1 rle with old method : 0.013265609741210938 time for calcul the mask position with numpy : 0.03252983093261719 nb_pixel_total : 429 time to create 1 rle with old method : 0.0006184577941894531 time for calcul the mask position with numpy : 0.025371551513671875 nb_pixel_total : 113319 time to create 1 rle with old method : 0.14567899703979492 time for calcul the mask position with numpy : 0.022448062896728516 nb_pixel_total : 76596 time to create 1 rle with old method : 0.09231233596801758 time for calcul the mask position with numpy : 0.02217245101928711 nb_pixel_total : 16278 time to create 1 rle with old method : 0.020889997482299805 time for calcul the mask position with numpy : 0.02933478355407715 nb_pixel_total : 135894 time to create 1 rle with old method : 0.17192649841308594 time for calcul the mask position with numpy : 0.037215471267700195 nb_pixel_total : 56314 time to create 1 rle with old method : 0.06243300437927246 time for calcul the mask position with numpy : 0.03536057472229004 nb_pixel_total : 24218 time to create 1 rle with old method : 0.030823707580566406 time for calcul the mask position with numpy : 0.036359548568725586 nb_pixel_total : 45391 time to create 1 rle with old method : 0.053029775619506836 time for calcul the mask position with numpy : 0.0364842414855957 nb_pixel_total : 4299 time to create 1 rle with old method : 0.005023479461669922 time for calcul the mask position with numpy : 0.036005496978759766 nb_pixel_total : 8388 time to create 1 rle with old method : 0.009611368179321289 time for calcul the mask position with numpy : 0.0366213321685791 nb_pixel_total : 10636 time to create 1 rle with old method : 0.01156163215637207 time for calcul the mask position with numpy : 0.03588509559631348 nb_pixel_total : 24952 time to create 1 rle with old method : 0.026897192001342773 time for calcul the mask position with numpy : 0.03542375564575195 nb_pixel_total : 127698 time to create 1 rle with old method : 0.1381533145904541 time for calcul the mask position with numpy : 0.03991985321044922 nb_pixel_total : 54137 time to create 1 rle with old method : 0.0596616268157959 time for calcul the mask position with numpy : 0.03666996955871582 nb_pixel_total : 81969 time to create 1 rle with old method : 0.09991693496704102 time for calcul the mask position with numpy : 0.03796553611755371 nb_pixel_total : 1663 time to create 1 rle with old method : 0.0021321773529052734 time for calcul the mask position with numpy : 0.03821873664855957 nb_pixel_total : 37716 time to create 1 rle with old method : 0.053075313568115234 time for calcul the mask position with numpy : 0.03866386413574219 nb_pixel_total : 18335 time to create 1 rle with old method : 0.020536422729492188 time for calcul the mask position with numpy : 0.03763079643249512 nb_pixel_total : 13121 time to create 1 rle with old method : 0.055463314056396484 time for calcul the mask position with numpy : 0.04095625877380371 nb_pixel_total : 179174 time to create 1 rle with new method : 0.7704222202301025 time for calcul the mask position with numpy : 0.042027950286865234 nb_pixel_total : 104247 time to create 1 rle with old method : 0.11865353584289551 create new chi : 3.762211561203003 time to delete rle : 0.0020644664764404297 batch 1 Loaded 23 chid ids of type : 3726 Number RLEs to save : 15524 TO DO : save crop sub photo not yet done ! save time : 2.725615978240967 nb_obj : 24 nb_hashtags : 7 time to prepare the origin masks : 13.159598588943481 time for calcul the mask position with numpy : 0.45833468437194824 nb_pixel_total : 6067658 time to create 1 rle with new method : 0.9303977489471436 time for calcul the mask position with numpy : 0.026343822479248047 nb_pixel_total : 7221 time to create 1 rle with old method : 0.008278846740722656 time for calcul the mask position with numpy : 0.04415273666381836 nb_pixel_total : 80207 time to create 1 rle with old method : 0.086334228515625 time for calcul the mask position with numpy : 0.039817094802856445 nb_pixel_total : 13975 time to create 1 rle with old method : 0.015475749969482422 time for calcul the mask position with numpy : 0.03802132606506348 nb_pixel_total : 19903 time to create 1 rle with old method : 0.023931503295898438 time for calcul the mask position with numpy : 0.03802227973937988 nb_pixel_total : 38777 time to create 1 rle with old method : 0.04494214057922363 time for calcul the mask position with numpy : 0.039037227630615234 nb_pixel_total : 36328 time to create 1 rle with old method : 0.0430145263671875 time for calcul the mask position with numpy : 0.0387113094329834 nb_pixel_total : 58195 time to create 1 rle with old method : 0.06325650215148926 time for calcul the mask position with numpy : 0.036325693130493164 nb_pixel_total : 29898 time to create 1 rle with old method : 0.03262782096862793 time for calcul the mask position with numpy : 0.04120683670043945 nb_pixel_total : 99873 time to create 1 rle with old method : 0.11059713363647461 time for calcul the mask position with numpy : 0.03692889213562012 nb_pixel_total : 69541 time to create 1 rle with old method : 0.07833361625671387 time for calcul the mask position with numpy : 0.0411527156829834 nb_pixel_total : 66135 time to create 1 rle with old method : 0.0718533992767334 time for calcul the mask position with numpy : 0.03788495063781738 nb_pixel_total : 27884 time to create 1 rle with old method : 0.04432845115661621 time for calcul the mask position with numpy : 0.03838396072387695 nb_pixel_total : 3479 time to create 1 rle with old method : 0.004053831100463867 time for calcul the mask position with numpy : 0.0513768196105957 nb_pixel_total : 2099 time to create 1 rle with old method : 0.00254058837890625 time for calcul the mask position with numpy : 0.03914690017700195 nb_pixel_total : 16498 time to create 1 rle with old method : 0.018470048904418945 time for calcul the mask position with numpy : 0.04614710807800293 nb_pixel_total : 9271 time to create 1 rle with old method : 0.010265588760375977 time for calcul the mask position with numpy : 0.04393434524536133 nb_pixel_total : 65702 time to create 1 rle with old method : 0.07146143913269043 time for calcul the mask position with numpy : 0.04363250732421875 nb_pixel_total : 2481 time to create 1 rle with old method : 0.002803325653076172 time for calcul the mask position with numpy : 0.04650545120239258 nb_pixel_total : 44334 time to create 1 rle with old method : 0.053456783294677734 time for calcul the mask position with numpy : 0.04007768630981445 nb_pixel_total : 8937 time to create 1 rle with old method : 0.009880781173706055 time for calcul the mask position with numpy : 0.039777278900146484 nb_pixel_total : 129373 time to create 1 rle with old method : 0.1398928165435791 time for calcul the mask position with numpy : 0.04373049736022949 nb_pixel_total : 9618 time to create 1 rle with old method : 0.011446237564086914 time for calcul the mask position with numpy : 0.04172539710998535 nb_pixel_total : 72751 time to create 1 rle with old method : 0.08066606521606445 time for calcul the mask position with numpy : 0.04035687446594238 nb_pixel_total : 70102 time to create 1 rle with old method : 0.07646870613098145 create new chi : 3.5216445922851562 time to delete rle : 0.0024344921112060547 batch 1 Loaded 25 chid ids of type : 3726 Number RLEs to save : 14579 TO DO : save crop sub photo not yet done ! save time : 2.576641321182251 nb_obj : 13 nb_hashtags : 3 time to prepare the origin masks : 9.488640069961548 time for calcul the mask position with numpy : 0.3479006290435791 nb_pixel_total : 5695162 time to create 1 rle with new method : 0.7838492393493652 time for calcul the mask position with numpy : 0.027948856353759766 nb_pixel_total : 36303 time to create 1 rle with old method : 0.04009842872619629 time for calcul the mask position with numpy : 0.03844332695007324 nb_pixel_total : 29943 time to create 1 rle with old method : 0.03432655334472656 time for calcul the mask position with numpy : 0.04114127159118652 nb_pixel_total : 950 time to create 1 rle with old method : 0.0011551380157470703 time for calcul the mask position with numpy : 0.04013562202453613 nb_pixel_total : 152 time to create 1 rle with old method : 0.00033164024353027344 time for calcul the mask position with numpy : 0.04166698455810547 nb_pixel_total : 75827 time to create 1 rle with old method : 0.08431816101074219 time for calcul the mask position with numpy : 0.04248404502868652 nb_pixel_total : 37958 time to create 1 rle with old method : 0.04217839241027832 time for calcul the mask position with numpy : 0.041091203689575195 nb_pixel_total : 42264 time to create 1 rle with old method : 0.04613971710205078 time for calcul the mask position with numpy : 0.04598093032836914 nb_pixel_total : 416826 time to create 1 rle with new method : 0.30330491065979004 time for calcul the mask position with numpy : 0.03840517997741699 nb_pixel_total : 246736 time to create 1 rle with new method : 0.9095757007598877 time for calcul the mask position with numpy : 0.03405499458312988 nb_pixel_total : 6652 time to create 1 rle with old method : 0.008907794952392578 time for calcul the mask position with numpy : 0.03261685371398926 nb_pixel_total : 176723 time to create 1 rle with new method : 0.9552645683288574 time for calcul the mask position with numpy : 0.03127121925354004 nb_pixel_total : 255916 time to create 1 rle with new method : 1.1858351230621338 time for calcul the mask position with numpy : 0.06049513816833496 nb_pixel_total : 28828 time to create 1 rle with old method : 0.03513741493225098 create new chi : 5.458065986633301 time to delete rle : 0.0019214153289794922 batch 1 Loaded 14 chid ids of type : 3726 Number RLEs to save : 11734 TO DO : save crop sub photo not yet done ! save time : 2.1428678035736084 nb_obj : 9 nb_hashtags : 5 time to prepare the origin masks : 7.117320537567139 time for calcul the mask position with numpy : 0.21677803993225098 nb_pixel_total : 3274758 time to create 1 rle with new method : 0.7517280578613281 time for calcul the mask position with numpy : 0.02622365951538086 nb_pixel_total : 86808 time to create 1 rle with old method : 0.09541058540344238 time for calcul the mask position with numpy : 0.03930544853210449 nb_pixel_total : 1394 time to create 1 rle with old method : 0.0018000602722167969 time for calcul the mask position with numpy : 0.043790340423583984 nb_pixel_total : 214701 time to create 1 rle with new method : 0.7851002216339111 time for calcul the mask position with numpy : 0.03329968452453613 nb_pixel_total : 6664 time to create 1 rle with old method : 0.007584571838378906 time for calcul the mask position with numpy : 0.03229689598083496 nb_pixel_total : 12702 time to create 1 rle with old method : 0.014624834060668945 time for calcul the mask position with numpy : 0.02425408363342285 nb_pixel_total : 43460 time to create 1 rle with old method : 0.04735565185546875 time for calcul the mask position with numpy : 0.026636362075805664 nb_pixel_total : 16393 time to create 1 rle with old method : 0.019640684127807617 time for calcul the mask position with numpy : 0.02263331413269043 nb_pixel_total : 36897 time to create 1 rle with old method : 0.04320883750915527 time for calcul the mask position with numpy : 0.18887686729431152 nb_pixel_total : 3356463 time to create 1 rle with new method : 0.8308999538421631 create new chi : 3.3595852851867676 time to delete rle : 0.0020279884338378906 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 12278 TO DO : save crop sub photo not yet done ! save time : 1.377373218536377 nb_obj : 15 nb_hashtags : 6 time to prepare the origin masks : 6.667954683303833 time for calcul the mask position with numpy : 0.427243709564209 nb_pixel_total : 6396268 time to create 1 rle with new method : 0.6560733318328857 time for calcul the mask position with numpy : 0.02165675163269043 nb_pixel_total : 992 time to create 1 rle with old method : 0.001486063003540039 time for calcul the mask position with numpy : 0.022154569625854492 nb_pixel_total : 50880 time to create 1 rle with old method : 0.05728268623352051 time for calcul the mask position with numpy : 0.03655076026916504 nb_pixel_total : 57551 time to create 1 rle with old method : 0.07492852210998535 time for calcul the mask position with numpy : 0.03547954559326172 nb_pixel_total : 4777 time to create 1 rle with old method : 0.0052487850189208984 time for calcul the mask position with numpy : 0.02910161018371582 nb_pixel_total : 18524 time to create 1 rle with old method : 0.0225980281829834 time for calcul the mask position with numpy : 0.024123191833496094 nb_pixel_total : 2291 time to create 1 rle with old method : 0.0025773048400878906 time for calcul the mask position with numpy : 0.0255124568939209 nb_pixel_total : 111500 time to create 1 rle with old method : 0.12987732887268066 time for calcul the mask position with numpy : 0.0225679874420166 nb_pixel_total : 2224 time to create 1 rle with old method : 0.0025565624237060547 time for calcul the mask position with numpy : 0.028536558151245117 nb_pixel_total : 3290 time to create 1 rle with old method : 0.0038230419158935547 time for calcul the mask position with numpy : 0.035083770751953125 nb_pixel_total : 80090 time to create 1 rle with old method : 0.10945296287536621 time for calcul the mask position with numpy : 0.03776431083679199 nb_pixel_total : 8478 time to create 1 rle with old method : 0.009333372116088867 time for calcul the mask position with numpy : 0.037099361419677734 nb_pixel_total : 164101 time to create 1 rle with new method : 0.975987434387207 time for calcul the mask position with numpy : 0.04366016387939453 nb_pixel_total : 51663 time to create 1 rle with old method : 0.14917707443237305 time for calcul the mask position with numpy : 0.15577268600463867 nb_pixel_total : 37406 time to create 1 rle with old method : 0.08652257919311523 time for calcul the mask position with numpy : 0.03954601287841797 nb_pixel_total : 60205 time to create 1 rle with old method : 0.16116738319396973 create new chi : 3.55159068107605 time to delete rle : 0.0030853748321533203 batch 1 Loaded 16 chid ids of type : 3726 Number RLEs to save : 9219 TO DO : save crop sub photo not yet done ! save time : 0.6623339653015137 nb_obj : 18 nb_hashtags : 6 time to prepare the origin masks : 11.734555959701538 time for calcul the mask position with numpy : 0.24307966232299805 nb_pixel_total : 6522861 time to create 1 rle with new method : 1.0084843635559082 time for calcul the mask position with numpy : 0.02860879898071289 nb_pixel_total : 63773 time to create 1 rle with old method : 0.07007360458374023 time for calcul the mask position with numpy : 0.029401540756225586 nb_pixel_total : 26806 time to create 1 rle with old method : 0.029805898666381836 time for calcul the mask position with numpy : 0.038185834884643555 nb_pixel_total : 4306 time to create 1 rle with old method : 0.005078554153442383 time for calcul the mask position with numpy : 0.04029130935668945 nb_pixel_total : 48125 time to create 1 rle with old method : 0.052041053771972656 time for calcul the mask position with numpy : 0.04421353340148926 nb_pixel_total : 35117 time to create 1 rle with old method : 0.03809380531311035 time for calcul the mask position with numpy : 0.04164385795593262 nb_pixel_total : 22334 time to create 1 rle with old method : 0.024553537368774414 time for calcul the mask position with numpy : 0.04401874542236328 nb_pixel_total : 1697 time to create 1 rle with old method : 0.002066373825073242 time for calcul the mask position with numpy : 0.04503989219665527 nb_pixel_total : 3861 time to create 1 rle with old method : 0.004537343978881836 time for calcul the mask position with numpy : 0.04126310348510742 nb_pixel_total : 4063 time to create 1 rle with old method : 0.004659175872802734 time for calcul the mask position with numpy : 0.04532623291015625 nb_pixel_total : 35750 time to create 1 rle with old method : 0.039307355880737305 time for calcul the mask position with numpy : 0.043311119079589844 nb_pixel_total : 279 time to create 1 rle with old method : 0.0005469322204589844 time for calcul the mask position with numpy : 0.0298917293548584 nb_pixel_total : 194337 time to create 1 rle with new method : 0.538804292678833 time for calcul the mask position with numpy : 0.03793191909790039 nb_pixel_total : 3432 time to create 1 rle with old method : 0.0039021968841552734 time for calcul the mask position with numpy : 0.022783994674682617 nb_pixel_total : 3095 time to create 1 rle with old method : 0.003392934799194336 time for calcul the mask position with numpy : 0.02383589744567871 nb_pixel_total : 15763 time to create 1 rle with old method : 0.01724410057067871 time for calcul the mask position with numpy : 0.026155710220336914 nb_pixel_total : 3795 time to create 1 rle with old method : 0.004273891448974609 time for calcul the mask position with numpy : 0.025746822357177734 nb_pixel_total : 29775 time to create 1 rle with old method : 0.03125882148742676 time for calcul the mask position with numpy : 0.025825023651123047 nb_pixel_total : 31071 time to create 1 rle with old method : 0.03285622596740723 create new chi : 2.839468240737915 time to delete rle : 0.0014524459838867188 batch 1 Loaded 19 chid ids of type : 3726 Number RLEs to save : 9203 TO DO : save crop sub photo not yet done ! save time : 1.1041815280914307 nb_obj : 16 nb_hashtags : 4 time to prepare the origin masks : 11.403990507125854 time for calcul the mask position with numpy : 1.0611908435821533 nb_pixel_total : 6720748 time to create 1 rle with new method : 1.3738353252410889 time for calcul the mask position with numpy : 0.027831315994262695 nb_pixel_total : 1458 time to create 1 rle with old method : 0.003003835678100586 time for calcul the mask position with numpy : 0.026903629302978516 nb_pixel_total : 20325 time to create 1 rle with old method : 0.033172607421875 time for calcul the mask position with numpy : 0.02930426597595215 nb_pixel_total : 2085 time to create 1 rle with old method : 0.004099130630493164 time for calcul the mask position with numpy : 0.05627608299255371 nb_pixel_total : 46236 time to create 1 rle with old method : 0.09333276748657227 time for calcul the mask position with numpy : 0.05003190040588379 nb_pixel_total : 8470 time to create 1 rle with old method : 0.013386726379394531 time for calcul the mask position with numpy : 0.04638099670410156 nb_pixel_total : 2166 time to create 1 rle with old method : 0.002669811248779297 time for calcul the mask position with numpy : 0.04182314872741699 nb_pixel_total : 156269 time to create 1 rle with new method : 1.0346555709838867 time for calcul the mask position with numpy : 0.04387688636779785 nb_pixel_total : 5962 time to create 1 rle with old method : 0.00681304931640625 time for calcul the mask position with numpy : 0.04805779457092285 nb_pixel_total : 1533 time to create 1 rle with old method : 0.0018503665924072266 time for calcul the mask position with numpy : 0.04561185836791992 nb_pixel_total : 34400 time to create 1 rle with old method : 0.25780177116394043 time for calcul the mask position with numpy : 0.0418546199798584 nb_pixel_total : 3112 time to create 1 rle with old method : 0.0037097930908203125 time for calcul the mask position with numpy : 0.041730642318725586 nb_pixel_total : 2585 time to create 1 rle with old method : 0.003009796142578125 time for calcul the mask position with numpy : 0.04193472862243652 nb_pixel_total : 16107 time to create 1 rle with old method : 0.017911672592163086 time for calcul the mask position with numpy : 0.04249215126037598 nb_pixel_total : 9392 time to create 1 rle with old method : 0.010651111602783203 time for calcul the mask position with numpy : 0.04533672332763672 nb_pixel_total : 6910 time to create 1 rle with old method : 0.00809478759765625 time for calcul the mask position with numpy : 0.04487919807434082 nb_pixel_total : 12482 time to create 1 rle with old method : 0.0162508487701416 create new chi : 4.703034400939941 time to delete rle : 0.0012946128845214844 batch 1 Loaded 17 chid ids of type : 3726 Number RLEs to save : 6903 TO DO : save crop sub photo not yet done ! save time : 0.8505902290344238 nb_obj : 14 nb_hashtags : 5 time to prepare the origin masks : 10.97117567062378 time for calcul the mask position with numpy : 0.24607539176940918 nb_pixel_total : 4401843 time to create 1 rle with new method : 0.9145026206970215 time for calcul the mask position with numpy : 0.03745555877685547 nb_pixel_total : 55117 time to create 1 rle with old method : 0.08228278160095215 time for calcul the mask position with numpy : 0.060552358627319336 nb_pixel_total : 267436 time to create 1 rle with new method : 0.7075521945953369 time for calcul the mask position with numpy : 0.05498313903808594 nb_pixel_total : 2408 time to create 1 rle with old method : 0.0028362274169921875 time for calcul the mask position with numpy : 0.05547761917114258 nb_pixel_total : 20322 time to create 1 rle with old method : 0.022704124450683594 time for calcul the mask position with numpy : 0.05455946922302246 nb_pixel_total : 7070 time to create 1 rle with old method : 0.009281396865844727 time for calcul the mask position with numpy : 0.05606269836425781 nb_pixel_total : 98243 time to create 1 rle with old method : 0.1085960865020752 time for calcul the mask position with numpy : 0.05453896522521973 nb_pixel_total : 67044 time to create 1 rle with old method : 0.07300281524658203 time for calcul the mask position with numpy : 0.03803277015686035 nb_pixel_total : 2784 time to create 1 rle with old method : 0.003375530242919922 time for calcul the mask position with numpy : 0.037670135498046875 nb_pixel_total : 34948 time to create 1 rle with old method : 0.04064822196960449 time for calcul the mask position with numpy : 0.03716444969177246 nb_pixel_total : 33430 time to create 1 rle with old method : 0.039201974868774414 time for calcul the mask position with numpy : 0.03796076774597168 nb_pixel_total : 116015 time to create 1 rle with old method : 0.1286458969116211 time for calcul the mask position with numpy : 0.0777590274810791 nb_pixel_total : 1893494 time to create 1 rle with new method : 0.9321744441986084 time for calcul the mask position with numpy : 0.04015851020812988 nb_pixel_total : 42270 time to create 1 rle with old method : 0.05142498016357422 time for calcul the mask position with numpy : 0.04103994369506836 nb_pixel_total : 7816 time to create 1 rle with old method : 0.00948333740234375 create new chi : 4.184321641921997 time to delete rle : 0.0021626949310302734 batch 1 Loaded 15 chid ids of type : 3726 Number RLEs to save : 14113 TO DO : save crop sub photo not yet done ! save time : 2.4366652965545654 nb_obj : 9 nb_hashtags : 4 time to prepare the origin masks : 8.356671571731567 time for calcul the mask position with numpy : 1.2534873485565186 nb_pixel_total : 6667259 time to create 1 rle with new method : 1.0242793560028076 time for calcul the mask position with numpy : 0.04576587677001953 nb_pixel_total : 24120 time to create 1 rle with old method : 0.05506443977355957 time for calcul the mask position with numpy : 0.0937645435333252 nb_pixel_total : 130805 time to create 1 rle with old method : 0.17276263236999512 time for calcul the mask position with numpy : 0.05699324607849121 nb_pixel_total : 8500 time to create 1 rle with old method : 0.016545772552490234 time for calcul the mask position with numpy : 0.05316591262817383 nb_pixel_total : 4276 time to create 1 rle with old method : 0.0056989192962646484 time for calcul the mask position with numpy : 0.05378985404968262 nb_pixel_total : 33296 time to create 1 rle with old method : 0.0448153018951416 time for calcul the mask position with numpy : 0.058768510818481445 nb_pixel_total : 9931 time to create 1 rle with old method : 0.011662006378173828 time for calcul the mask position with numpy : 0.061295270919799805 nb_pixel_total : 46584 time to create 1 rle with old method : 0.05517172813415527 time for calcul the mask position with numpy : 0.05521535873413086 nb_pixel_total : 9565 time to create 1 rle with old method : 0.01401209831237793 time for calcul the mask position with numpy : 0.05319809913635254 nb_pixel_total : 115904 time to create 1 rle with old method : 0.146697998046875 create new chi : 3.407602071762085 time to delete rle : 0.0019390583038330078 batch 1 Loaded 10 chid ids of type : 3726 Number RLEs to save : 8588 TO DO : save crop sub photo not yet done ! save time : 0.7532732486724854 map_output_result : {1335192933: (0.0, 'Should be the crop_list due to order', 0.0), 1335192926: (0.0, 'Should be the crop_list due to order', 0.0), 1335192920: (0.0, 'Should be the crop_list due to order', 0.0), 1335192136: (0.0, 'Should be the crop_list due to order', 0.0), 1335192010: (0.0, 'Should be the crop_list due to order', 0.0), 1335192000: (0.0, 'Should be the crop_list due to order', 0.0), 1335191995: (0.0, 'Should be the crop_list due to order', 0.0), 1335191988: (0.0, 'Should be the crop_list due to order', 0.0), 1335191949: (0.0, 'Should be the crop_list due to order', 0.0), 1335191945: (0.0, 'Should be the crop_list due to order', 0.0)} End step rle-unique-nms Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : rle_unique_nms_with_priority we use saveGeneral [1335192933, 1335192926, 1335192920, 1335192136, 1335192010, 1335192000, 1335191995, 1335191988, 1335191949, 1335191945] Looping around the photos to save general results len do output : 10 /1335192933.Didn't retrieve data . /1335192926.Didn't retrieve data . /1335192920.Didn't retrieve data . /1335192136.Didn't retrieve data . /1335192010.Didn't retrieve data . /1335192000.Didn't retrieve data . /1335191995.Didn't retrieve data . /1335191988.Didn't retrieve data . /1335191949.Didn't retrieve data . /1335191945.Didn't retrieve data . before output type Used above Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192933', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192926', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192920', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192136', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192010', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192000', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191995', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191988', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191949', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191945', None, None, None, None, None, '2560885') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.015012264251708984 save_final save missing photos in datou_result : time spend for datou_step_exec : 165.50144267082214 time spend to save output : 0.03171586990356445 total time spend for step 5 : 165.5331585407257 step6:crop_condition Fri Feb 7 01:51:31 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure some photos are not treated, begin crop_condition Loading chi in step crop with photo_hashtag_type : 3726 Loading chi in step crop for list_pids : 10 ! batch 1 Loaded 173 chid ids of type : 3726 begin to crop the class : teint_dans_la_masse param for this class : {'min_score': 0.7} filtre for class : teint_dans_la_masse hashtag_id of this class : 2107752385 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 9 About to insert : list_path_to_insert length 9 new photo from crops ! About to upload 9 photos upload in portfolio : 4869462 init cache_photo without model_param we have 9 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738889495_972799 we have uploaded 9 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.4377212524414062 we have finished the crop for the class : teint_dans_la_masse begin to crop the class : autre_refus param for this class : {'min_score': 0.5} filtre for class : autre_refus hashtag_id of this class : 2107752406 Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 10 About to insert : list_path_to_insert length 10 new photo from crops ! About to upload 10 photos upload in portfolio : 4869462 init cache_photo without model_param we have 10 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738889503_972799 we have uploaded 10 photos in the portfolio 4869462 time of upload the photos Elapsed time : 2.9313416481018066 we have finished the crop for the class : autre_refus begin to crop the class : carton_gris param for this class : {'min_score': 0.5} filtre for class : carton_gris hashtag_id of this class : 2107753020 Next one ! Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 4 About to insert : list_path_to_insert length 4 new photo from crops ! About to upload 4 photos upload in portfolio : 4869462 init cache_photo without model_param we have 4 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738889510_972799 we have uploaded 4 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.2825305461883545 we have finished the crop for the class : carton_gris begin to crop the class : cartonnette param for this class : {'min_score': 0.5} filtre for class : cartonnette hashtag_id of this class : 702398920 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738889514_972799 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.2929041385650635 we have finished the crop for the class : cartonnette begin to crop the class : carton_brun param for this class : {'min_score': 0.7} filtre for class : carton_brun hashtag_id of this class : 2107753024 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738889517_972799 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 1.0909020900726318 we have finished the crop for the class : carton_brun begin to crop the class : plastique param for this class : {'min_score': 0.5} filtre for class : plastique hashtag_id of this class : 492725882 Next one ! Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 3 About to insert : list_path_to_insert length 3 new photo from crops ! About to upload 3 photos upload in portfolio : 4869462 init cache_photo without model_param we have 3 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738889521_972799 we have uploaded 3 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.8816578388214111 we have finished the crop for the class : plastique begin to crop the class : kraft param for this class : {'min_score': 0.5} filtre for class : kraft hashtag_id of this class : 493202403 Next one ! Next one ! map_result returned by crop_photo_return_map_crop : length : 2 About to insert : list_path_to_insert length 2 new photo from crops ! About to upload 2 photos upload in portfolio : 4869462 init cache_photo without model_param we have 2 photo to upload uploaded to storage server : ovh folder_temporaire : temp/1738889523_972799 we have uploaded 2 photos in the portfolio 4869462 time of upload the photos Elapsed time : 0.7248015403747559 we have finished the crop for the class : kraft begin to crop the class : metal param for this class : {'min_score': 0.5} filtre for class : metal hashtag_id of this class : 492628673 delete rles for these photos Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : crop_condition we use saveGeneral [1335192933, 1335192926, 1335192920, 1335192136, 1335192010, 1335192000, 1335191995, 1335191988, 1335191949, 1335191945] Looping around the photos to save general results len do output : 33 /1335333153Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333154Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333155Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333156Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333157Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333158Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333159Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333160Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333161Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333181Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333182Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333183Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333184Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333186Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333187Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333188Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333190Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333191Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333192Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333201Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333203Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333204Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333205Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333214Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333215Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333222Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333223Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333224Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333228Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333229Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333230Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333235Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . /1335333236Didn't retrieve data .Didn't retrieve data .Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192933', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192926', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192920', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192136', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192010', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192000', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191995', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191988', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191949', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191945', None, None, None, None, None, '2560885') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 109 time used for this insertion : 0.030903100967407227 save_final save missing photos in datou_result : time spend for datou_step_exec : 35.544557332992554 time spend to save output : 0.0442051887512207 total time spend for step 6 : 35.588762521743774 step7:ventilate_hashtags_in_portfolio Fri Feb 7 01:52:07 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure beginning of datou step ventilate_hashtags_in_portfolio : To implement ! To do loadFromThcl(), then load ParamDescType : thcl2725 thcls : [{'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'}] thcl {'id': 2725, 'mtr_user_id': 31, 'name': 'learn_qualipapia_rle_210302_2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Carton_brun,Carton_gris,Teint_Dans_La_Masse,autre_refus,cartonnette,environnement,kraft,metal,papier,plastique', 'svm_portfolios_learning': '3460440,3460441,3460446,3460434,3460439,3467416,3460442,3460443,3486028,3460445', 'photo_hashtag_type': 3410, 'photo_desc_type': 5186, 'type_classification': 'caffe', 'hashtag_id_list': '2107753024,2107753020,2107752385,2107752406,702398920,493012381,493202403,492628673,492668766,492725882'} Update svm_hashtag_type_desc : 5186 Iterating over portfolio : 20314827 get user id for portfolio 20314827 SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314827 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('environnement','Teint_Dans_La_Masse','papier','mal_croppe','Carton_brun','autre_refus','kraft','plastique','flou','cartonnette','Carton_gris','metal')) AND mptpi.`min_score`=0.5 To do To do ! Use context local managing function ! SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314827 AND mptpi.`type`=3726 AND mptpi.`hashtag_id` in (select hashtag_id FROM MTRBack.hashtags where hashtag in ('environnement','Teint_Dans_La_Masse','papier','mal_croppe','Carton_brun','autre_refus','kraft','plastique','flou','cartonnette','Carton_gris','metal')) AND mptpi.`min_score`=0.5 To do lien utilise dans velours : https://www.fotonower.com/velours/20314926,20314927,20314928,20314929,20314930,20314931,20314932,20314933,20314934,20314935,20314936,20314937?tags=environnement,Teint_Dans_La_Masse,flou,kraft,metal,mal_croppe,Carton_gris,papier,Carton_brun,autre_refus,cartonnette,plastique Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : ventilate_hashtags_in_portfolio we use saveGeneral [1335192933, 1335192926, 1335192920, 1335192136, 1335192010, 1335192000, 1335191995, 1335191988, 1335191949, 1335191945] Looping around the photos to save general results len do output : 1 /20314827. before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192933', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192926', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192920', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192136', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192010', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192000', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191995', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191988', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191949', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191945', None, None, None, None, None, '2560885') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.014083623886108398 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.878204345703125 time spend to save output : 0.014412641525268555 total time spend for step 7 : 0.8926169872283936 step8:final Fri Feb 7 01:52:08 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 VR 22-3-18 : For now we do not clean correctly the datou structure Beginning of datou step final ! Catched exception ! Connect or reconnect ! Inside saveOutput : final : False verbose : 0 original output for save of step final : {1335192933: ('0.08185036069797114',), 1335192926: ('0.08185036069797114',), 1335192920: ('0.08185036069797114',), 1335192136: ('0.08185036069797114',), 1335192010: ('0.08185036069797114',), 1335192000: ('0.08185036069797114',), 1335191995: ('0.08185036069797114',), 1335191988: ('0.08185036069797114',), 1335191949: ('0.08185036069797114',), 1335191945: ('0.08185036069797114',)} new output for save of step final : {1335192933: ('0.08185036069797114',), 1335192926: ('0.08185036069797114',), 1335192920: ('0.08185036069797114',), 1335192136: ('0.08185036069797114',), 1335192010: ('0.08185036069797114',), 1335192000: ('0.08185036069797114',), 1335191995: ('0.08185036069797114',), 1335191988: ('0.08185036069797114',), 1335191949: ('0.08185036069797114',), 1335191945: ('0.08185036069797114',)} [1335192933, 1335192926, 1335192920, 1335192136, 1335192010, 1335192000, 1335191995, 1335191988, 1335191949, 1335191945] Looping around the photos to save general results len do output : 10 /1335192933.Didn't retrieve data . /1335192926.Didn't retrieve data . /1335192920.Didn't retrieve data . /1335192136.Didn't retrieve data . /1335192010.Didn't retrieve data . /1335192000.Didn't retrieve data . /1335191995.Didn't retrieve data . /1335191988.Didn't retrieve data . /1335191949.Didn't retrieve data . /1335191945.Didn't retrieve data . before output type Used above Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192933', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192926', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192920', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192136', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192010', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192000', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191995', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191988', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191949', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191945', None, None, None, None, None, '2560885') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 30 time used for this insertion : 0.013288021087646484 save_final save missing photos in datou_result : time spend for datou_step_exec : 0.12090158462524414 time spend to save output : 0.014018535614013672 total time spend for step 8 : 0.1349201202392578 step9:velours_tree Fri Feb 7 01:52:08 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 VR 22-3-18 : For now we do not clean correctly the datou structure can't find the photo_desc_type Inside saveOutput : final : False verbose : 0 ouput is None No outpout to save, returning out of save general time spend for datou_step_exec : 0.2983112335205078 time spend to save output : 5.91278076171875e-05 total time spend for step 9 : 0.298370361328125 step10:send_mail_cod Fri Feb 7 01:52:08 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! complete output_args for input 2 Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 3 We should have FATAL ERROR but same_nb_input_output==True : this should be an optionnal input ! VR 22-3-18 : For now we do not clean correctly the datou structure dans la step send mail cod work_area: /home/admin/workarea/git/Velours/python in order to get the selector url, please entre the license of selector results_Auto_P20314827_07-02-2025_01_52_08.pdf 20314928 imagette203149281738889528 20314929 change filename to text .change filename to text .change filename to text .imagette203149291738889528 20314930 imagette203149301738889528 20314931 imagette203149311738889528 20314935 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203149351738889528 20314936 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203149361738889529 20314937 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203149371738889530 20314927 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203149271738889532 20314932 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203149321738889533 20314934 change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .change filename to text .imagette203149341738889534 SELECT h.hashtag,pcr.value FROM MTRUser.portfolio_carac_ratio pcr, MTRBack.hashtags h where pcr.portfolio_id=20314827 and hashtag_type = 3726 and pcr.hashtag_id = h.hashtag_id; velour_link : https://www.fotonower.com/velours/20314926,20314927,20314928,20314929,20314930,20314931,20314932,20314933,20314934,20314935,20314936,20314937?tags=environnement,Teint_Dans_La_Masse,flou,kraft,metal,mal_croppe,Carton_gris,papier,Carton_brun,autre_refus,cartonnette,plastique your option no_mail is active, we will not send the real mail to your client args[1335192933] : ('0.08185036069797114',) no score found for photo 1335192933 apple ('0.08185036069797114',) We are sending mail with results at report@fotonower.com args[1335192926] : ('0.08185036069797114',) no score found for photo 1335192926 apple ('0.08185036069797114',) We are sending mail with results at report@fotonower.com args[1335192920] : ('0.08185036069797114',) no score found for photo 1335192920 apple ('0.08185036069797114',) We are sending mail with results at report@fotonower.com args[1335192136] : ('0.08185036069797114',) no score found for photo 1335192136 apple ('0.08185036069797114',) We are sending mail with results at report@fotonower.com args[1335192010] : ('0.08185036069797114',) no score found for photo 1335192010 apple ('0.08185036069797114',) We are sending mail with results at report@fotonower.com args[1335192000] : ('0.08185036069797114',) no score found for photo 1335192000 apple ('0.08185036069797114',) We are sending mail with results at report@fotonower.com args[1335191995] : ('0.08185036069797114',) no score found for photo 1335191995 apple ('0.08185036069797114',) We are sending mail with results at report@fotonower.com args[1335191988] : ('0.08185036069797114',) no score found for photo 1335191988 apple ('0.08185036069797114',) We are sending mail with results at report@fotonower.com args[1335191949] : ('0.08185036069797114',) no score found for photo 1335191949 apple ('0.08185036069797114',) We are sending mail with results at report@fotonower.com args[1335191945] : ('0.08185036069797114',) no score found for photo 1335191945 apple ('0.08185036069797114',) We are sending mail with results at report@fotonower.com refus_total : 0.08185036069797114 2022-04-13 10:29:59 0 SELECT ph.photo_id,ph.url,ph.username,ph.uploaded_at,ph.text FROM MTRBack.photos ph, MTRUser.mtr_portfolio_photos mpp WHERE ph.photo_id=mpp.mtr_photo_id AND mpp.mtr_portfolio_id=20314827 AND mpp.hide_status=0 ORDER BY mpp.order LIMIT 0, 1000 SELECT photo_id, url FROM MTRBack.photos ph WHERE photo_id IN (1335192933,1335193060,1335193055,1335193052,1335193049,1335193011,1335193003,1335192939,1335192938,1335192936,1335191693,1335192926,1335192136,1335191995,1335191988,1335191945,1335191696,1335192920,1335192010,1335192999) Found this number of photos: 20 begin to download photo : 1335192933 begin to download photo : 1335193011 begin to download photo : 1335191693 begin to download photo : 1335191945 download finish for photo 1335191693 begin to download photo : 1335192926 download finish for photo 1335191945 begin to download photo : 1335191696 download finish for photo 1335193011 begin to download photo : 1335193003 download finish for photo 1335192933 begin to download photo : 1335193060 download finish for photo 1335192926 begin to download photo : 1335192136 download finish for photo 1335193060 begin to download photo : 1335193055 download finish for photo 1335191696 begin to download photo : 1335192920 download finish for photo 1335193003 begin to download photo : 1335192939 download finish for photo 1335192920 begin to download photo : 1335192010 download finish for photo 1335192136 begin to download photo : 1335191995 download finish for photo 1335192939 begin to download photo : 1335192938 download finish for photo 1335193055 begin to download photo : 1335193052 download finish for photo 1335192010 begin to download photo : 1335192999 download finish for photo 1335191995 begin to download photo : 1335191988 download finish for photo 1335192938 begin to download photo : 1335192936 download finish for photo 1335191988 download finish for photo 1335192999 download finish for photo 1335192936 download finish for photo 1335193052 begin to download photo : 1335193049 download finish for photo 1335193049 start upload file to ovh https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314827_07-02-2025_01_52_08.pdf results_Auto_P20314827_07-02-2025_01_52_08.pdf uploaded to url https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314827_07-02-2025_01_52_08.pdf start insert file to database insert into MTRUser.mtr_files (mtd_id,mtr_portfolio_id,text,url,format,tags,file_size,value) values ('3459','20314827','results_Auto_P20314827_07-02-2025_01_52_08.pdf','https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314827_07-02-2025_01_52_08.pdf','pdf','','1.12','0.08185036069797114') Inside saveOutput : final : False verbose : 0 saveOutput not yet implemented for datou_step.type : send_mail_cod we use saveGeneral [1335192933, 1335192926, 1335192920, 1335192136, 1335192010, 1335192000, 1335191995, 1335191988, 1335191949, 1335191945] Looping around the photos to save general results len do output : 0 before output type Used above Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192933', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192926', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192920', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192136', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192010', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192000', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191995', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191988', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191949', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191945', None, None, None, None, None, '2560885') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 10 time used for this insertion : 1.2404217720031738 save_final save missing photos in datou_result : time spend for datou_step_exec : 11.352844715118408 time spend to save output : 1.2707178592681885 total time spend for step 10 : 12.623562574386597 step11:split_time_score Fri Feb 7 01:52:21 2025 VR 17-11-17 : now, only for linear exec dependencies tree, some output goes to fill the input of the next VR 22-3-18 : now we test the dependencies tree, but keep two separate code for datou_prepare_output_input until the code is correctly tested, clean and works in both case VR 22-3-18 : but we use the first code for the first step id = -1, build in the code of datou_exec VR 22-3-18 : we should manage here the case when we are at the first step instead of building this step before datou_exec Currently we do not manage missing dependencies information, that could maybe be correctly interpreted with default behavior Some of the step done at execution of the step could be done before when the tree of execution is build and the dependencies of different step analysed complete output_args for input 0 We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array We expect there is only one output and this part is used while all output are not tuple or array Inconsistent number of input and output, step which parrallelize and manage error in input by avoiding sending an output for this data can't be used in tree dependencies of input and output complete output_args for input 1 VR 22-3-18 : For now we do not clean correctly the datou structure begin split time score Catched exception ! Connect or reconnect ! TODO : Insert select and so on Begin split_port_in_batch_balle thcls : [{'id': 861, 'mtr_user_id': 31, 'name': 'Rungis_class_dechets_1212', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': 'Rungis_Aluminium,Rungis_Carton,Rungis_Papier,Rungis_Plastique_clair,Rungis_Plastique_dur,Rungis_Plastique_fonce,Rungis_Tapis_vide,Rungis_Tetrapak', 'svm_portfolios_learning': '1160730,571842,571844,571839,571933,571840,571841,572307', 'photo_hashtag_type': 999, 'photo_desc_type': 3963, 'type_classification': 'caffe', 'hashtag_id_list': '2107751280,2107750907,2107750908,2107750909,2107750910,2107750911,2107750912,2107750913'}] thcls : [{'id': 758, 'mtr_user_id': 31, 'name': 'Rungis_amount_dechets_fall_2018_v2', 'pb_hashtag_id': 0, 'live': b'\x00', 'list_hashtags': '05102018_Papier_non_papier_dense,05102018_Papier_non_papier_peu_dense,05102018_Papier_non_papier_presque_vide,05102018_Papier_non_papier_tres_dense,05102018_Papier_non_papier_tres_peu_dense', 'svm_portfolios_learning': '1108385,1108386,1108388,1108384,1108387', 'photo_hashtag_type': 856, 'photo_desc_type': 3853, 'type_classification': 'caffe', 'hashtag_id_list': '2107751013,2107751014,2107751015,2107751016,2107751017'}] (('10', 22),) ERROR counted https://github.com/fotonower/Velours/issues/663#issuecomment-421136223 {} 06022025 20314827 Nombre de photos uploadées : 22 / 23040 (0%) 06022025 20314827 Nombre de photos taguées (types de déchets): 0 / 22 (0%) 06022025 20314827 Nombre de photos taguées (volume) : 0 / 22 (0%) elapsed_time : load_data_split_time_score 6.67572021484375e-06 elapsed_time : order_list_meta_photo_and_scores 2.0742416381835938e-05 ?????????????????????? elapsed_time : fill_and_build_computed_from_old_data 0.0016329288482666016 elapsed_time : insert_dashboard_record_day_entry 0.04069328308105469 We will return after consolidate but for now we need the day, how to get it, for now depending on the previous heavy steps find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314812 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314813 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314814 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314815 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314816 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314817 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314818 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314819 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314820 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314821 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314822 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314823 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314824 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314825 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314826 order by id desc limit 1 Qualite : 0.07702728454025964 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314827_07-02-2025_01_52_08.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314827 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314827 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314828 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314829 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314830 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314831 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314832 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314833 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314834 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314835 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314836 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314837 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314838 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314839 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314840 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314841 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314842 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314843 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314844 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314845 order by id desc limit 1 Qualite : 0.09628085696972587 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314846_07-02-2025_01_37_46.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314846 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314846 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314847 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314848 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314849 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314850 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314851 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314852 order by id desc limit 1 Qualite : 0.08573655308919938 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314853_07-02-2025_01_44_23.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314853 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 11449 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 11452 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Step 11452 crop_condition have less outputs used (2) than in the step definition (3) : some outputs may be not used ! Step 11453 merge_mask_thcl_custom have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11453 merge_mask_thcl_custom is not consistent : 4 used against 2 in the step definition ! WARNING : number of inputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11454 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! Step 11478 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 11478 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 11456 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 11455 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 11455 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! Step 11458 send_mail_cod have less inputs used (3) than in the step definition (5) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! WARNING : type of output 2 of step 11449 doesn't seem to be define in the database( WARNING : type of input 2 of step 11452 doesn't seem to be define in the database( WARNING : output 1 of step 11449 have datatype=2 whereas input 1 of step 11453 have datatype=7 WARNING : type of output 2 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11454 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 3 of step 11453 doesn't seem to be define in the database( WARNING : type of input 1 of step 11456 doesn't seem to be define in the database( WARNING : type of output 1 of step 11456 doesn't seem to be define in the database( WARNING : type of input 3 of step 11455 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 11456 have datatype=10 whereas input 3 of step 11458 have datatype=6 WARNING : type of input 5 of step 11458 doesn't seem to be define in the database( WARNING : output 0 of step 11477 have datatype=11 whereas input 5 of step 11458 have datatype=None WARNING : output 0 of step 11456 have datatype=10 whereas input 0 of step 11477 have datatype=18 WARNING : type of input 2 of step 11478 doesn't seem to be define in the database( WARNING : output 1 of step 11454 have datatype=7 whereas input 2 of step 11478 have datatype=None WARNING : type of output 3 of step 11478 doesn't seem to be define in the database( WARNING : type of input 2 of step 11456 doesn't seem to be define in the database( WARNING : output 0 of step 11453 have datatype=1 whereas input 0 of step 11454 have datatype=2 DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314853 AND mptpi.`type`=3726 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314854 order by id desc limit 1 Qualite : 0.2245652913943355 find url: https://storage.sbg.cloud.ovh.net/v1/AUTH_3b171620e76e4af496c5fd050759c9f0/media.fotonower.com/results_Auto_P20314855_07-02-2025_01_50_41.pdf select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314855 order by id desc limit 1 # VR 17-11-17 : to create in DB ! Here we check the datou graph and we reorder steps ! Tree builded and cycle checked, now we need to re-order the steps ! We have currenlty an error because there is no dependence between the last step for the case tile - detect - glue We can either keep the depence of, it is better to keep an order compatible with the id of steps if we do not have sons, so a lexical order : (number_son, step_id) All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! All sons are already in current list ! DONE and to test : checkNoCycle ! Here we check the consistency of inputs/outputs number between the given ones and the db ! eke 1-6-18 : checkConsistencyNbInputNbOutput should be processed after step reordering ! WARNING : number of outputs for step 7928 mask_detect is not consistent : 3 used against 2 in the step definition ! Step 8092 crop_condition have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! WARNING : number of outputs for step 8092 crop_condition is not consistent : 4 used against 3 in the step definition ! WARNING : number of inputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7933 rle_unique_nms_with_priority is not consistent : 2 used against 1 in the step definition ! WARNING : number of outputs for step 7935 ventilate_hashtags_in_portfolio is not consistent : 2 used against 1 in the step definition ! Step 7934 final have less inputs used (2) than in the step definition (3) : maybe we manage optionnal inputs ! Step 7934 final have less outputs used (1) than in the step definition (2) : some outputs may be not used ! WARNING : number of outputs for step 13649 velours_tree is not consistent : 2 used against 1 in the step definition ! Step 9283 split_time_score have less inputs used (1) than in the step definition (2) : maybe we manage optionnal inputs ! Number of inputs / outputs for each step checked ! Here we check the consistency of outputs/inputs types during steps connections eke 1-6-18 : checkConsistencyTypeOutputInput should be processed after checkConsistencyNbInputNbOutput ! We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of output 1 of step 7935 doesn't seem to be define in the database( WARNING : type of input 3 of step 7934 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : type of input 1 of step 7935 doesn't seem to be define in the database( WARNING : output 1 of step 7933 have datatype=7 whereas input 1 of step 7935 have datatype=None WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 2 of step 8092 doesn't seem to be define in the database( WARNING : type of output 3 of step 8092 doesn't seem to be define in the database( WARNING : type of input 1 of step 7933 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10917 doesn't seem to be define in the database( WARNING : type of output 2 of step 7928 doesn't seem to be define in the database( WARNING : type of input 1 of step 10918 doesn't seem to be define in the database( We ignore checkConsistencyTypeOutputInput for datou_step final ! WARNING : output 0 of step 7935 have datatype=10 whereas input 3 of step 10916 have datatype=6 WARNING : output 0 of step 7935 have datatype=10 whereas input 0 of step 13649 have datatype=18 WARNING : type of output 1 of step 13649 doesn't seem to be define in the database( WARNING : type of input 5 of step 10916 doesn't seem to be define in the database( DataTypes for each output/input checked ! TODO Duplicate data, are they consistent 3 ? Duplicate data, are they consistent 4 ? SELECT mptpi.id, mptpi.mtr_portfolio_id_1, mptpi.mtr_portfolio_id_2, mptpi.type, mptpi.hashtag_id, mptpi.min_score, mptpi.mtr_user_id, mptpi.created_at, mptpi.updated_at, mptpi.last_updated_at_desc, mptpi.last_updated_at_asc, h.hashtag FROM MTRPhoto.mtr_port_to_port_ids mptpi, MTRBack.hashtags h WHERE h.hashtag_id=mptpi.hashtag_id AND mptpi.`mtr_portfolio_id_1`=20314855 AND mptpi.`type`=3594 To do find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314856 order by id desc limit 1 find url: select completion_json, dashboard_run_id from MTRPhoto.dashboard_results where mtr_portfolio_id = 20314857 order by id desc limit 1 NUMBER BATCH : 0 # DISPLAY ALL COLLECTED DATA : {'06022025': {'nb_upload': 22, 'nb_taggue_class': 0, 'nb_taggue_densite': 0}} Inside saveOutput : final : True verbose : 0 saveOutput not yet implemented for datou_step.type : split_time_score we use saveGeneral [1335192933, 1335192926, 1335192920, 1335192136, 1335192010, 1335192000, 1335191995, 1335191988, 1335191949, 1335191945] Looping around the photos to save general results len do output : 1 /20314827Didn't retrieve data . before output type Here is an output not treated by saveGeneral : Managing all output in save final without adding information in the mtr_datou_result ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192933', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192926', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192920', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192136', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192010', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335192000', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191995', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191988', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191949', None, None, None, None, None, '2560885') ('3459', None, None, None, None, None, None, None, '2560885') ('3459', None, '1335191945', None, None, None, None, None, '2560885') begin to insert list_values into mtr_datou_result : length of list_values in save_final : 11 time used for this insertion : 0.015017032623291016 save_final save missing photos in datou_result : time spend for datou_step_exec : 32.99856638908386 time spend to save output : 0.015328168869018555 total time spend for step 11 : 33.01389455795288 caffe_path_current : About to save ! 2 After save, about to update current ! update_current_state 174.90user 143.33system 6:32.21elapsed 81%CPU (0avgtext+0avgdata 4516324maxresident)k 1917272inputs+151616outputs (38331major+14208300minor)pagefaults 0swaps